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I.  INTRODUCTION 


The  objective  of  the  research  which  culminated  in  this  set  of 
documents  was  to  develop  a process  for  the  procurement  of  two 
evaluation  systems.  Within  the  context  of  applicable  procurement 
regulations  and  program  plans,  the  set  provides  a complete  and 
integrated  framework  for  both  the  preparation  of  proposals  by 
prospective  bidders  and  the  evaluation  of  proposals  by  the  eval- 
uation team. 

The  document  entitled  Proposal  Preparation  Procedure  provides 
both  an  overall  description  of  the  process  and  specific  guidance 
to  prospective  bidders.  All  of  the  salient  or  significant 
characteristics  of  the  system  to  be  procured,  the  program  under 
which  it  will  be  developed,  and  supporting  elements  are  included 
in  a set  of  elements  called  sub-factors  which  are  grouped,  for 
convenience,  into  a set  of  factors.  Each  of  the  key  documents-- 
the  Statement  of  Work,  Schedule,  Scope  of  Effort,  and  Evaluation 
Factor s - -follows  a parallel  organization  of  the  set  of  sub-factors. 
It  is  this  integration  of  all  of  the  central  documents  which 
provides  both  proposer  and  evaluator  with  a single,  common  struc- 
ture . 
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The  initiation  of  the  development  of  this  process  arose  out 
of  the  recognition  that  the  "evaluation  systems"  to  be  procured 
required  the  services  of  a systems  contractor  and  that  it  appear- 
ed that  neither  ETIP  nor  the  "evaluation  industry"  had  any 
significant  present  capability  to  design  or  manage  such  a system. 

A review  of  available  proposal  preparation  and  evaluation  pro- 
cedures suggested  that  some  further  development  was  required. 

The  process  described  herein  was  used  to  procure  the  two 
systems,  and,  as  of  the  initial  getting  started  period  under  the 
two  contracts,  appears  to  have  worked  out  reasonably  well  in  terms 
of  our  confidence  that  we  have  chosen  not  only  the  best  but  also 
reasonably  competent  contractors. 

While  this  set  of  documents  was  specifically  designed  for  the 
procurement  of  these  two  evaluation  systems,  the  process,  with 
modifications  in  the  set  of  sub-factors,  is  applicable  to  compar- 
able systems  problems  involving  complex  or  uncertain  requirements. 
One  of  the  purposes  of  this  report  is  to  make  these  documents 
available  on  the  basis  that  others  may  be  interested  in  further 
applications . 
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Introductory  Notes 

The  purpose  of  these  two  proposed  procurements  is  to  obtain 
contractor  assistance  in  planning,  designing,  and  carrying 
out  the  evaluation  of  a series  of  procurement  experiments 
which  are  being  undertaken  by  ETIP  in  cooperation  with  several 
governmental  procurement  agencies. 

The  set  of  objectives  of  the  evaluation  program  includes  the 
following: 

a.  an  overall  description  of  the  experiments  as  well  as 
selected  detailed  descriptions 

b.  an  overall  assessment  of  both  the  immediate  and  the 
subsequent  effects  (impacts)  of  the  experiments,  as 
well  as  selected  detailed  assessments 

c.  An  evaluation  system  or  process  which  can  be  used  by 
the  appropriate  government  agency  (ies)  to  obtain 
evaluations  on  a continuing  basis  of  these  as  well 
as  future,  similar  experiments. 

There  are  a number  of  choices  which  could  be  made  in  obtaining 
contractor  assistance,  ranging  from  a single  overall  contractor 
to  individual  contractors  for  each  experiment,  or  combinations 
in  between.  The  choice  which  has  been  made  is  to  divide  the 
overall  evaluation  program  into  two  procurements,  as  follows: 

a.  an  evaluation  of  "agency  impact"  which  will  direct 
its  attention  to  a description  of  the  experiments, 

an  assessment  of  the  effects  on  the  several  government 
agencies,  and  the  design  of  the  related  evaluation 
system.  "Agency  impact"  includes  both  immediate  and 
subsequent  effects  on  the  program  administrative  agency 
but  also  other  affected  or  related  government  agencies 

b.  an  evaluation  of  "commercial  impact"  which  will  direct 
its  attention  to  the  immediate  and  subsequent  effects 
upon  the  commercial  (or  industrial)  sector,  and,  as 
appropriate,  consumers  or  users  of  the  products,  and 
the  design  of  the  related  evaluation  system. 


1 


RFP  6-35756 


The  set  of  "experiments"  includes  the  following: 

a.  specific  procurement  experiments  - including 
"completed,"  on-going,  and  planned 

b.  specific  related  or  supportive  experiments  - includ- 
ing procedures  for  selecting  experiments,  training 
of  agency  personnel,  processes  for  obtaining  from 

or  providing  to  the  industrial  sector  information 
required  for  carrying  out  procurement  experiments, 
and  other  administrative  changes 

c.  general  "experimental"  changes  - related  to  or  in 
support  of  the  above. 

The  set  of  experiments  may  be  conveniently  grouped  into  broad 
categories  according  to  these  dimensions: 

a.  by  program  administrative  agency  - (1)  Federal 

Supply  Services  (FSS) ; (2)  state  and  local  procure- 

ment agencies  (S&L) ; and  (3)  Veterans  Administration 
(VA) 

b.  by  type  of  procurement  experiment  (1)  life  cycle 

costing  (LCC) ; (2)  value  incentive  contracting  (VIC) ; 

(3)  performance  specifications;  (4)  multi-year  awards; 
etc . 

c.  by  the  kind  of  effect  intended  - (1)  economic  or  per- 
formance advantage  to  the  using  government  agency; 

(2)  related  or  derivative  improvement  in  procurement 
capabilities  of  the  program  administrative  agency; 

(3)  increased  or  improved  technological  innovation 

in  the  industrial  sector;  and  (4)  economic  or  perfor- 
mance advantage  to  civilian  sector  users. 

To  obtain  the  evaluation  desired  we  propose  to  let  two  contracts, 
each  for  a three-year  period,  divided  into  three  phases,  as 
follows : 

Phase  One  - to  include,  generally,  three  activities 

(a)  preliminary  systems  analysis 

(b)  evaluation  of  specific,  selected  procurement  or 
other  experiments 

(c)  design  and  pilot  test  of  the  related  evaluation 
system 
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Phase  Two  - to  include , generally: 

(a)  refinement  of  (a)  above 

(b)  evaluation  of  additional  specific  procurement 
or  other  experiments 

(c)  refinement  and  prototype  test  of  the  related 
evaluation  system 

Phase  Three  - to  include,  generally,  "turn  key"  implementa- 
tion of  the  related  evaluation  system  by 
the  appropriate  government  agency 

The  complete  procurement  package  has  been  designed  to  assure  the 
fullest  possible  interchange  of  information  between  prospective 
bidders  and  those  responsible  for  the  evaluation  of  proposals. 
Because  of  this  it  will  be  necessary  to  examine  the  complete 
set;  particular  attention  is  directed  to  the  fact  that  the 
Statement  of  Work  and  the  Schedule  must  be  read  in  conjunction 
with  the  accompanying  documents. 
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Some  Definitions 


Note:  In  most  cases,  the  context,  or  reference  to  other  parts 

of  the  bid  set  and  associated  documents,  should  resolve 
ambiguities  or  otherwise  clarify  definitions  of  terms.  These 
"definitions"  are  provided  as  a convenience  in  reference. 

ETIP  - Refers  to  the  Experimental  Technology  Incentives  Pro- 
gram. In  context,  the  reference  may  be  to  the  objectives  or 
content  of  the  program,  to  the  office  or  organization  itself 
(e.g.,  as  contracting  agency),  or  to  individual  staff  members. 

PAA  - Refers  to  the  Program  Administrative  Agency  (or  Agencies)  . 
There  are,  for  these  procurements  (agency  impact  and  commercial 
impact),  three  PAA,  as  follows:  1)  Federal  Supply  Service 

(FSS) ; 2)  state  and  local  procurement  agencies  (S£L)  ; 

3)  Veterans'  Administration  (VA) . In  context,  the  reference 
will  usually  be  to  the  specific  procurement  office (s)  and/or 
responsible  personnel  in  the  agency  which  have  the  direct 
authority  and/or  responsibility  for  the  decision  to  intro- 
duce the  specific  experimental  intervention  and,  usually, 
for  the  conduct  of  the  program  and/or  function  to  which  the 
experimental  condition  is  applied.  For  FSS,  this  may  include 
both  central  and  regional  procurement  functions  or  offices. 

For  SSL , this  may  include  NASPO,  NIGP,  and/or  specific  state 
and  local  procurement  agencies.  For  VA  this  may  include  both 
central  and  regional  or  local  procurement  agencies. 

AGENCY  IMPACT  - Refers  to  effects  or  impacts  on  or  in  the  PAA 
and,  where  applicable,  closely  related  governmental  agencies, 
and,  particularly,  agencies  which  are  the  users  of  the  products 
procured.  These  impacts  or  effects  include  not  only  those 
which  are  the  specific  objectives  of  a specific  or  general 
experiment,  but  also  significant  other  effects,  such  as 
changes  in  administrative  policies  and  procedures. 

COMMERCIAL  IMPACT  - Refers  to  effects  upon  the  commercial 
sector  and/or  civilian  sector  users.  These  impacts  or  effects 
include  not  only  those  which  are  the  specific  objectives  of  a 
specific  or  general  experiment,  but  also  significant  other 
effects,  such  as  increased  or  earlier  use  of  new  technology. 

EXPERIMENT  - Refers,  generically,  to  any  combination  of 
intervention  and  desired  (or  hypothesized)  effect.  In  a 
complex  of  interventions  and  effects,  there  may  be  a number 
of  "experiments,"  including  experiments  which  are  combina- 
tions of  other  experiments,  and  experiments  where  the  effect 
of  one  is  the  intervention  of  another. 
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SPECIFIC  EXPERIMENT  - Refers,  usually,  to  a specific,  identified 
intervention  and  desired  (or  hypothesized)  effect.  The  major, 
and  most  significant,  cases  are  the  specific  procurement  experi- 
ments (e.g.,  use  of  LCC  to  procure  air  conditioners).  There 
may  be  other  (related)  specific  experiments  (e.g.,  the  FSS  train- 
ing program) . 

VARIABLE  - Refers,  generically,  to  any  object,  event,  state, 
value,  function,  etc.  In  the  context  of  a specific  experiment 
(or  study),  the  independent  variable  is,  usually,  the  inter- 
vention, and  the  dependent  variable  the  desired  (or  hypothesized) 
effect . 

PARAMETER  - Refers  to  those  variables  which,  whether  controllable 
(directly  or  indirectly)  or  not,  affect  (or  are  hypothesized 
to  affect)  plausibly  and/or  significantly  the  interpretability 
and/or  credibility  of  the  observed  (or  hypothesized)  relation- 
ship between  the  variables  in  a specific  experiment. 

A PRIORI  PROPOSITION  TESTING  - Refers  to  those  specific  experi- 
ments for  which  one  of,  if  not  the  primary,  purposes  is  to 
establish,  with  a relatively  high  degree  of  credibility,  the 
relationship  between  the  intervention  and  the  effect  observed. 

In  practice,  this  will  be  largely  limited  to  relatively  well 
defined  interventions  and  proximal  effects,  and  where 
reasonable  control  of  parameters  can  be  achieved. 

EXPLORATORY  (AND/OR  DESCRIPTIVE)  - Refers  to  experiments  (and/ 
or  studies)  where  it  is  either  not  feasible  or  not  required 
to  meet  the  requirements  of  a priori  proposition  testing.  In 
practice,  this  will  apply  to  requirements  to  describe  and/or 
measure  sets  of  variables  where  either  the  description  is 
sufficient  itself  or  provides  a basis  for  identifying  hypotheses, 
variables,  and/or  parameters. 

EVALUATION  SYSTEM  - Refers  to  the  set  of  policies  and  procedures 
which  provides  the  basis  for  evaluating  the  (agency  and/or 
commercial)  impacts  of  a set  of  specific  procurement  or  related 
experiments.  The  form,  detail  and  completeness  of  the  system 
will  progressively  change.  In  Phase  One,  it  may  begin  as  a 
preliminary  outline  within  which  detailed  "single  thread" 
designs  are  developed  for  the  early  evaluation  of  specific 
procurement  experiments;  later,  the  results  of  the  preliminary 
systems  analysis  and  the  "pilot  test"  will  provide  the  basis 
for  a preliminary  evaluation  systems  design.  In  Phase  Two, 
progressive  refinement  should  result  in  a relatively  complete 
evaluation  system  which  can  be  tested  as  a prototype,  in  part, 
through  the  evaluation  of  specific  experiments.  In  Phase 
Three,  the  evaluation  system  should  be  in  the  form  of  a stable 
and  complete  system. 
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BASIC  (OR  BASE-LINE)  EVALUATION  PROCESS  - Refers  to  that  part 
of  the  evaluation  system  which  includes  those  evaluation 
activities  which  deal  with  key,  predictable,  identifiable, 
continuing,  common  objectives  for  which  the  process  can  be 
institutionalized . 

SPECIAL  EVALUATION  PROCESS  - Refers  to  those  evaluation  activi- 
ties which  do  not  meet  the  above  requirements  because  of 
specialized,  one-time  or  changing  objectives. 

FRAMEWORK  OR  MODEL  - Refers  to  graphical  or  other  conceptual 
representation  of  the  set  of  interrelations  among  objectives, 
organizations,  experiments,  and/or  programs,  and  the  evalua- 
tion system. 
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PROPOSAL  PREPARATION  PROCEDURE 


A.  Purpose 

The  purpose  of  this  document  is  to  provide  a guide  for 
bidders  in  the  preparation  of  their  proposals. 

B . Basis  for  Preparation  of  This  Document 
B.l  Historical  Note 

A wide  variety  of  methods  have  been  used  for  the  procurement 
of  the  services  of  evaluation  contractors,  including  the 
following:  1)  adding  evaluation  as  an  express  or  implied 

requirement  to  the  basic  contract  for  the  program  or  experi- 
ment; 2)  contracting  separately  with  a contractor  or  consultant 
for  the  evaluation;  3)  obtaining  an  evaluation,  usually  post 
hoc , as  part  of  the  study  phase  of  a contract  for  a new  or 
different  program.  The  description  of  the  services  desired 
may  take  many  forms,  including  the  following:  1)  a brief, 

general  requirement  "to  perform  an  evaluation";  2)  state- 
ments or  descriptions  of  various  lengths  and  with  various 
degrees  of  detail  and  completeness  which  outline  the  object- 
ives of  the  program  and/or  the  specific  questions  to  be 
answered;  3)  relatively  detailed  and  complete  "specifications" 
of  the  services  required,  including  instruments  to  be  used, 
sampling  plans,  and  the  form  of  the  analysis  required. 

In  some  cases,  and  for  some  purposes,  these  methods  provide 
a satisfactory  base  for  defining  the  services  required, 
which,  in  turn,  may  provide  a satisfactory  basis  for  determ- 
ining the  qualifications  of  prospective  contractors. 

None  of  the  above  appear  appropriate  as  a basis  for  determin- 
ing the  qualifications  of  prospective  contractors  on  this 
procurement.  For  this  reason  this  document  was  prepared, 
using  a "parametric  factor  analytic  approach"  which  was 
developed  for  and  used  on  a number  of  subsystem  procurements 
by  another  agency  of  the  government. 
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B.2  Basis  for  Evaluation  (of  Bidders) 

Generally,  evaluation  (including  proposals)  requires  three 
things:  first,  a standard  or  base  for  comparing;  second, 

information  to  be  compared  against  the  standard;  and,  third, 
an  effective  method  for  doing  the  comparing.  These  are 
discussed  in  the  three  paragraphs  below. 

B.2.1  The  first  requirement  of  the  evaluation  of  competitive 
proposals  is  a standard  or  base  for  comparing,  and  this  is, 
obviously  "who  can  do  the  job  best."  but  this  is  a matter  of 
predicting  because  it  is  a future  matter,  and  there  is 
uncertainty  in  specifying  the  job.  The  choice  of  standard 
for  selection  includes: 

a.  The  whole  program,  either  as  it  is  or  as  it  should 
be  stated  to  obtain  the  selection  base. 

b.  The  "significant  independent  variable"  - if  one 
part  will  determine,  either  by  its  absence  or 
presence,  the  one  bidder  who  can  do  it,  then 
this  is  the  basis. 

c.  Some  sample  of  parameters  (variables  or  "factors") 
which  on  a presence  or  absence  basis,  or  qualitative 
scale,  is  the  determinant. 

The  choice  is  between  b)  and  c)  because  of  the  inability  to 
precisely  define  the  skills  and  content  of  the  future 
completely;  and  because  agreement  on  b)  presumes  considerable 
confidence  in  past  experience,  the  basis  must  be  c) . The 
development  of  c)  is  summarized  in  the  paragraph  below  in 
terms  of  a brief  statement  of  the  program. 

The  nature  of  the  item  to  be  procured  determines  the  program. 

The  evaluation  here  is  not  limited  to  fully  designed  evalua- 
tions of  specific  experiments,  nor  is  the  alternative  of  a 
broad,  overall  descriptive  case  study  sufficient.  It  is 
proposed  to  obtain  not  only  preliminary  systems  study  which 
will  include  both  of  the  above,  but  also  both  kinds  of 
evaluations;  in  subsequent  phases,  it  is  proposed  to 
develop  and  test  an  ongoing  capability  to  carry  out  such 
evaluations  in  such  form  and  detail  as  will  allow  the 
government  to  implement  subsequent  phases.  Direct 
description  of  such  a program  must  necessarily  be  accomplished 
by  selecting  and  tabulating  the  significant  parts  of  the 
program;  and  these  parts  may  then  be  further  defined  and 
amplified  by  supporting  data.  These  parts,  identified  as 
"factors,"  are  the  bases  for  the  establishment  of  the 
standard;  and,  in  turn,  the  information  to  be  furnished 
and  the  evaluation  procedures. 
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B.2.2  The  information  to  be  compared  against  the  standard 
is  that  furnished  by,  or  about,  the  bidders.  This  requires 
cooperation,  varying  from  a little  to  a lot.  Too  little 
occurs  when  the  bidder's  data  is  absent  or  cannot  be  trans- 
lated into  a common  dimension  or  is  irrelevant.  Too  much 
occurs  when  the  bidder  essentially  repeats  back  the  guide- 
lines furnished.  The  problem  of  "too  much"  has  not  been 
experienced  in  prior  evaluations,  and  those  few  cases  of 
"parroting"  were  obvious.  The  concern  is  to  assure  that 
the  bidder  has  thorough  guidance  in  preparing  the  necessary 
information,  and  by  this  means  the  bidder  will  be  aided 
in  directing  his  efforts  more  efficiently. 

B.2.3  Because  of  the  large  number  of  factors  to  be  evalu- 
ated, the  large  volume  of  data  to  be  considered,  and  the 
size  of  the  evaluation  team,  the  process  needs  to  be  well 
organized  and  supported  with  efficient  and  clear  procedures. 
Forms  and  instructions  must  be  prepared,  and  arrangements 
for  scheduling  meetings  and  for  carrying  out  the  evalua- 
tion have  to  be  thought  out  in  advance.  A corollary  benefit 
is  an  increase  in  the  assurance  of  impartiality  through 
avoidance  of  the  confusion  of  setting  rules  and  judging 
at  the  same  time. 

B.3  Preparation 

This  document,  and  the  associated  statement  of  work,  was 
prepared  upon  the  above  basis.  As  source  material, 
procedures  similar  to  this,  as  well  as  a number  of 
other  evaluation  procedures,  were  reviewed. 
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C . General  Comments  on  Proposal  Evaluation  Procedure 

To  provide  bidders  with  a general  background  and  perspective, 
a brief  outline  of  the  overall  procedure  is  furnished  below. 

(It  should  be  noted  that  these  comments  are  intended  only  as 
a general  description,  and  some  changes  may  be  made.) 

C.l  Preparation  for  Proposal  Evaluation 

C.1.1  To  assure  continuity,  personnel  responsible  for  setting 
up  and  administering  the  program  for  which  proposals  are  being 
solicited,  will  conduct  the  proposal  evaluation,  with  the 
advice  and  assistance  of  others  who  are  less  directly  con- 
cerned. Direct  management  responsibility  is  centered  in  ETIP 
in  the  Contracting  Officer's  Technical  Representative  (COTR) . 
He,  together  with  other  members  of  the  ETIP  staff,  will  be 
working  closely  with  key  members  of  the  staff  of  the  program 
administrative  agency  in  the  type  of  close  relationship 
which  is  characteristic  of  "administrative  experimentation." 

As  a minimum,  it  is  intended  that  at  least  two  members  of 
the  proposal  evaluation  team  will  be  assigned  primary  re- 
sponsibility for  each  subfactor,  with  backup  assistance  in 
review  by  other  team  members.  These  evaluation  team  members 
are  responsible  for  preparing  and/or  reviewing  the  bases 
for  the  evaluation  and  for  briefing  the  bidders. 

C.1.2  The  results  of  the  preparation  are  incorporated, 
primarily,  in  the  RFP.  In  addition  to  the  Form  33  and 
accompanying  provisions,  the  RFP  includes  a series  of 
related  documents  prepared  for  the  primary  purpose  of 
furnishing  a base  for  the  preparation  of  proposals.  T’he 
necessary  clarification  and  revision  to  provide  a contract- 
ual basis  will  be  accomplished  by  negotiation  with  the 
selected  contractor.  Briefly,  these  documents  are 
grouped  as  follows: 

Statement  of  Work  - tabulates  the  significant  parts 
of  the  program  with  clarification  limited  primarily 
to  a brief  summary.  This  will  include  those  separate- 
ly identifiable  "deliverables"  and  a schedule. 

Proposal  Preparation  Procedure  (this  document)  - provides 
an  explanation  of  the  basis  for  proposal  preparation. 

Proposal  Evaluation  Factors  - provides  the  basis  for 
proposal  preparation. 

Scope  of  Effort  - cost  proposal  requirements 

Other  Documents  - development  plans,  reports,  references 
etc.,  to  assist  the  bidders  in  preparing  their  proposals. 
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C. 1.2.1  To  the  extent  possible,  all  of  these  documents  follow 
the  same  order.  The  Statement  of  Work  includes  items  of 
required  work  divided  into  three  phases.  The  Proposal 
Preparation  Procedure  is  designed  to  match  the  organization 
of  the  Statement  of  Work.  It  is  intended  to  provide,  where 
necessary,  cross-reference  to  the  other  documents. 

C.l.2.2  To  assure  that  all  bidders  have  complete  sets  of  the 
related  documents,  and  especially  those  included  under 
"Others"  which  may  be  furnished  at  the  briefing,  a complete 
index  of  all  documents  will  also  be  furnished  during  the 
briefings,  as  required. 

C.1.3  It  is  planned  to  brief  the  bidders  twice.  At  the 
initial  briefing,  a summary  presentation  on  the  program 
will  be  given,  including  a presentation  by  the  program 
administrative  agency  on  the  administrative  objectives 
for  the  program.  (To  assure  the  continuity  specified  above, 
representatives  of  the  program  administrative  agency  will 
participate  in  the  evaluation.)  Following  the  presentation, 
an  extensive  question  period  will  be  scheduled.  The  second 
briefing  will  follow  a few  weeks  later  after  the  bidders 
have  had  an  opportunity  to  study  the  program;  this  is  primarily 
to  allow  for  further  questioning  by  the  bidders  to  correct 
oversights  and  resolve  ambiguities.  No  information  will 
be  available  during  the  proposal  preparation  period  on  an 
individual  basis. 

C.2  Proposed  Evaluation  Procedure 

C.2.1  After  receipt  of  the  written  proposals,  the  proposal 
evaluation  team  will  have  a period  of  a few  days  to 
familiarize  themselves  generally  with  the  proposals;  and, 
specifically,  the  parts  within  their  responsibility.  This 
is  primarily  to  furnish  a basis  for  the  next  stage. 

C.2. 2 According  to  a schedule  to  be  established  at  the  second 
of  the  two  briefings  noted  above,  each  bidder  will  be  invited 
to  make  an  oral  presentation,  approximately  one  hour  in 
length,  following  which  the  evaluation  team  will  have  two  to 
four  hours  for  questioning.  Purpose  of  the  questioning  is 
to  assure  that  the  evaluation  team  understands  the  proposals, 
and  has  an  adequate  basis  for  evaluation  on  each  sub-factor. 
Both  weak  and  strong  points  will  be  explored,  and  check  lists 
will  be  recorded  to  assure  that  team  members  may  have  available 
to  them  both  the  written  and  oral  presentations. 
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C.2.3  Following  the  above  will  come  a further  period  of 
review,  during  which  individual  team  members  will  complete 
their  individual  evaluations  of  sub-factors  for  which  they 
are  responsible  or  concerned.  These  individual  evaluations 
will  be  noted  with  the  assistance  of  a check  list  based  on 
a value  rating  scale  to  avoid  the  confusion  caused  by 
numerical  rating. 

C.2.4  Members  of  the  proposal  evaluation  team  concerned  with 
each  factor  will  meet  and  perform  a joint  evaluation.  They 
will  follow  an  established  procedure  entitled,  "Factor  Team 
Guidelines,"  to  assure  an  orderly  and  efficient  process. 

Briefly,  they  will  first  compare  notes  informally  to  assure 
common  standards  and  information;  second,  they  will  arrange 
the  bidders  in  descending  order;  third,  they  will  establish 
the  relative  spread  among  the  bidders;  and,  finally,  their 
result  will  be  permanently  recorded  and  signed  by  all  team 
members  present.  This  record  becomes  part  of  the  permanent 
evaluation  files.  Generally,  they  will  rate  on  each  sub- 
factor, as  applicable,  three  central  points: 

a.  Comprehension  or  understanding  of  the  problem 
presented 

b.  Proposed  approach  (or,  in  the  alternative,  the 
means  for  establishing  the  approach) 

c.  Present  or  potential  capability  to  accomplish  this 

Evaluation  team  members  may  use  information  obtained  from  the 
RFP  and  associated  documents,  the  bidders'  written  and  oral 
proposals  and  independent  or  prior  information.  However,  in 
the  latter  case,  any  significant  or  critical  information  must 
be  considered  by  all  members  of  the  factor  team.  Emphasis 
will  be  placed  on  the  sub-factor  being  rated,  but  the  impact 
of  other  factors  will  not  be  ignored.  Comparative  weighting 
of  individual  sub-factors  and  the  weight  of  each  factor  team 
members ' s views  will  be  within  the  responsibility  of  the 
factor  team. 

C.2.5  The  results  of  the  factor  team  evaluations  will  be 
summarized  and  reviewed. 

C.2.5.1  The  summary  process  will  include  the  necessary  combining 
of  the  individual  factor  ratings  to  achieve  an  overall  rating, 
with  consideration  of  the  interaction  of  factors,  their  indi- 
vidual significance  (acceptability) , and  the  "summation" 
according  to  previously  established  weights. 
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C.2.5.2  In  general,  the  range  of  weights  among  the  factors 
will  be  no  more  than  2:1. 

C.2.5.3  The  review  process  will  necessarily  provide  for 
latitude  in  reflecting  the  results  of  the  review,  such  as 
reassessing  of  weights  or  reexamination  of  specific  areas 
of  the  evaluation;  and,  such  changes  will  be  incorporated, 
with  the  supporting  basis  in  the  record  of  the  evaluation. 

C.3  Comment  on  Factor  and  Sub-factor  Structure 

All  of  the  factors  in  the  evaluation  are  set  out  in  the 
Proposal  Evaluation  Factors  documents,  and  are  largely 
self-explanatory.  The  rationale  of  this  particular 
organization  of  the  sample  is  based,  primarily,  on  the 
sub-factors,  the  factors  being  administrative  groupings 
for  convenience  in  organization  of  the  various  documents 
and  the  evaluation.  Generally,  the  factor/sub-factor  organiza- 
tion is  based  on  significant  identifiable  effort/capability 
areas.  The  intent  is  to  select  comprehensive  exclusive/ 
inclusive  samples  of  the  total  program.  In  total,  the 
factor/sub-factors  are  intended  to  include  samples  of  all 
significant  parts  of  the  program.  An  effort  has  been  made 
to  minimize  duplication;  and,  with  a few  intentional 
exceptions,  no  sub-factor  samples  an  area  covered  by  another 
factor.  Similarly,  with  certain  intentional  exceptions, 
each  sub-factor  samples  on  the  basis  of  understanding, 
approach,  and  capability;  as  a result,  those  aspects,  which 
include  "effective  experience,"  are  incorporated  into  specific 
substantive  areas  rather  than  amorphous  generalized  categories. 
It  is  important  to  understand  that  the  distinction  among  the 
factors  is  primarily  an  administrative  distinction  to  reflect 
the  differences  in  perspective  required  in  presentation  and 
evaluation . 


13 


RFP  6-35756 


D.  Use  of  this  Document 


D.l  General  Considerations 

D.1.1  As  noted  above,  the  Proposal  Evaluation  Factors  docu- 
ment covers  all  of  the  factors  to  be  considered  in  the 
evaluation  and  are  intended  to  provide  a substantially 
complete  framework  for  the  preparation  of  the  proposal. 
Because  the  background  and  perspective  against  which  they 
were  prepared  is  set  out  in  this  part,  the  other  related 
documents,  and  the  briefings  for  the  bidders,  it  is 
extremely  important  that  this  interpolation  be  understood 
and  preserved  during  the  preparation  period  by  assuring  that 
substantive  contributors  have  sufficient  access  to  the 
overall  framework.  As  a general  guide,  cross  reference 
should  be  made  to  the  applicable  points  included  in  the 
document  on  Scope  of  Effort. 

D.2  Content  Considerations 

D.2.1  Because  the  evaluation  is  organized  on  a factor/sub- 
factor basis,  it  is  important  that  the  treatment  of  each 
factor  be  substantially  complete  as  possible.  The  several 
factors  are  necessarily  interrelated  and  interacting,  but 
repetition  is  not  required;  however,  where  significant 
material  considered  necessary  to  assure  understanding  of 
a factor  is  contained  in  another  factor,  specific  (and 
annotated)  cross-references  should  be  furnished.  All  members 
of  the  evaluation  team  will  have  access  to  the  complete 
proposal,  and  it  is  expected  that  certain  factors  will  be 
reviewed  jointly. 

D.2. 2 Each  factor  and  sub-factor  is  defined  by  a specific 
descriptive  sentence  setting  out  the  standard  against 
which  each  bidder  will  be  measured.  Because  the  factor 
statements  reflect  the  fact  that  the  factor  is  primarily 
an  administrative  convenience,  no  specific  presentation 
with  regard  to  the  overall  factor  is  required.  Each  sub- 
factor, however,  will  be  the  subject  of  a specific  rating, 
and  the  proposal  content  should  reflect  the  necessity  for 
furnishing  the  evaluation  team  with  sufficient  information 
to  establish  a comparative  position  on  each  sub-factor 
THE  BASIC  ORGANIZATION  OF  THE  PROPOSAL  MUST  BE  ON  A SUB- 
FACTOR BASIS. 
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D.2.2.1  It  should  be  specifically  noted  that  the  sub-paragraphs 
under  each  sub-factor  are  provided  for  the  purpose  of  amplifying 
and  defining  the  standard,  that  is,  to  present  the  types  of 
questions  which  need  to  be  answered  to  establish  the  rating  on 
the  sub-factor.  It  is  not  required  to  answer  each  and  all  of 
these  subsidiary  questions.  Nor  is  it  required  that  the 
presentation  be  limited  to  these  particular  subsidiary  questions. 
In  some  cases,  a subsidiary  question  will  be  a critical  de- 
terminant; in  other  cases,  an  effective  presentation  may  be 
accomplished  by  partial  treatment.  It  was  not  intended  to 
specify  the  proposal  content  through  these  subsidiary  questions. 

D.2.2.2  These  subsidiary  questions  are  presented  in  three  forms. 
One  form  is  that  of  a question,  such  as,  "Do  you  consider  the 
proposed  schedule  realistic?  too  tight?  If  not,  what  do  you 
propose  and  why?"  The  second  general  form  is  that  of  a state- 
ment calling  for  comment  or  the  furnishing  of  certain  informa- 
tion, such  as,  "Testing  facilities  (floor  space,  equipment, 
personnel)."  The  third  general  form  is  that  of  a statement 
commenting  on  a problem  area  or  outlining  a proposed  solution 
which  furnishes  a basis  for  comment,  such  as,  "The  Advisory 
Committee  may  raise  a technical  question  which  requires 
investigation  in  order  to  allow  the  committee  to  make  a decision 
at  the  next  meeting.  The  contractor  will  furnish  the  necessary 
personnel  to  develop  each  of  the  conflicting  points  of  view 
independently. " 

D.2.2.3  As  noted  elsewhere,  the  relative  weighting,  or  signifi- 
cance, of  each  sub-factor  within  a factor  is  within  discretion 
of  the  factor  ream  performing  that  part  of  the  evaluation.  In 
most  cases,  the  sub-factors  represent  approximately  equivalent 
areas  of  significance.  The  amount  of  subsidiary  questions 
furnished  does  not  necessarily  reflect  an  unusually  high  or 
low  weight. 

D.2.3  Latitude  is  encouraged  in  the  treatment  of  the  content 
presented  with  respect  to  the  sub-factors.  Where  alternate 
assumptions  or  conclusions  to  clarify  ambiguities  or  fully 
establish  the  position  is  considered  advisable,  no  arbitrary 
limitation  is  imposed.  The  substantive  objective  is  to  provide 
information  as  a basis  for  evaluation,  and  this  procedure 
is  intended  only  as  a minimum  framework  to  increase  the 
efficiency  and  effectiveness  of  the  proposing  and  evaluating 
process . 

D.3  Physical  Considerations 

D.3.1  Primary  concern  with  regard  to  the  physical  format  and 
organization  should  be  to  facilitate  the  work  of  the  evalua- 
tion team,  recognizing  the  circumstances  and  procedures  which 
will  be  used. 
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D.3.2  It  is  suggested  that  the  proposal  use  the  same  major 
numerical  and  subject  headings  of  the  Proposal  Evaluation 
Factors  document  to  avoid  the  confusion  of  an  additional 
numbering  system.  This  should  be  extended  only  to  the 
factors  and  sub-factors,  and  no  requirement  to  follow  the 
numerical  order  of  the  subsidiary  questions  within  a sub- 
factor is  intended.  Where  it  is  considered  desirable  to 
include  additional  material  within  the  framework  of  Sections 
I through  VII,  an  expanded  numbering  sequence  should  be 
used,  i.e.,  to  include  material  after  1.1,  but  before  1.1.1, 
use  a new  number,  1.1.0,  with  subsidiary  numbers,  1. 1.0.1, 

I . 1 . 0 . 2 , etc . 

D.3.3  To  facilitate  the  physical  handling  of  the  proposal, 
certain  physical  divisions  are  requested. 

D.3.3.1  Physically  bulky  material,  such  as  extensive 
personnel  or  facility  brochures,  or  supporting  descriptive 
documents  furnished  in  amplification,  should  be  treated  as 
physically  separate  appendices,  with  suitable  identification. 
In  this  regard,  where  material  which  has  been  prepared  in 
another  connection  is  furnished,  a suitable  cover  sheet 
noting  limitations  of  application  may  be  used  to  avoid 
extensive  revision  and  republication. 

D.3.4  Provision  should  be  made  for  sufficient  indexing  and 
cross-reference  summaries  to  facilitate  reference  to 
particular  parts  of  the  proposal.  Unless  considered 
desirable,  pagination  need  not  be  accomplished.  It  is 
recommended  that  covers  of  separate  documents  contain 
clear  reference  to  the  numerical  and  subject  heading. 
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REVISIONS  TO  PROPOSAL  PREPARATION  PROCEDURES 


A.  Page  5.  -Delete  C.2.2,  and  substitute  the  following: 

C.2.2  Where  factor  team  members  require  specific 
additional  information  for  their  evaluation,  specific 
written  questions  will  be  directed  to  the  specific 
proposer  through  the  contracting  officer. 

B.  Page  6,  C.2.3,  lines  6 and  7.  Place  period  (.)  after 
"scales,"  and  delete  the  remaining  material. 


C.  Page  6,  C.2.4,  lines  24  to  27.  Delete  "Comparative 
weighting"  and  add  "Comparisons,"  and  delete  "weight" 
and  add  "efforts." 
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(INCLUDING  RELATION  TO  EVALUATIONS  OF  SPECIFIC  EXPERIMENTS) 
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Statement  of  Work 


Note:  The  items  listed  below  provide  a catagorized  summary  of  the 

services  and  materials  required  on  this  program. 

Phase  One:  Planning  and  Design,  and  preliminary  evaluation. 

Item  1.  Plan,  organize,  staff,  direct,  and  control,  including 
both  schedule  and  costs. 


Item  2.  Provide  reports,  liaison,  and  services  for  purposes 
of  review,  coordination,  and  approvals  and  accept- 
ances, including  the  following: 

a.  Periodic  and  special  formal  and  informal 
management  progress  reports,  final  report. 

b.  Periodic  and  final  scheduling  reports, 
including  projected  schedule  for  the  next  phase. 

c.  Periodic  and  final  cost  reports,  including 
projected  costs  for  the  next  phase. 

d.  Periodic  and  special  formal  and  informal 
technical  progress  reports  and  final  report, 
including  proposed  program  for  the  next  phase. 
(Note:  the  substantive  work  to  be  reported 
here  is  to  be  performed  under  items  3 through 
7)  . 


Item  3.  Review  and  structure  the  proposed  objectives  in 
terms  of  the  experimental  setting  and  related 
background,  and  establish  an  overall  plan  and 
set  of  priorities. 


Item  4.  Review  and  structure  the  proposed  set  of  experiments, 
and  related  background  in  terms  of  the  proposed 
objectives,  and  provide  a reference  base. 

Item  5.  Develop  the  detailed  evaluation  requirements  (problem 
definition  and  experimental  or  study  design) , in 
relation  to  collection  and  analysis. 


Item  6.  Develop  the  detailed  data  collection  process, 
including  instruments,  identification  of,  and 
access  to,  sources,  and  managing  the  process; 
and  collect  the  data. 
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Item  7:  Develop  the  detailed  data  analysis  process; 

carry  out  the  analyses;  present  findings; 
present  recommendations. 

Phase  Two:  Prototype  Evaluation  Process  Testing 


Item  8. 

Same  as  Item  1. 

Item  9. 

Same  as  Item  2,  but  change  reference  to  Items 
10  through  14. 

Item  10. 

Refine  and  revise  the  work  accomplished  under 
Item  3,  as  related  to  the  further  objective  of 
designing  and  testing  a prototype  of  the  complete 
evaluation  process. 

Item  11. 

Refine  and  revise  the  work  accomplished  under 
Item  4,  as  related  to  the  further  objective  of 
preparing  a formal  framework  within  which  the 
objectives,  organizations,  and  programs  may  be 
related  to  the  evaluation  process. 

Item  12. 

Refine  and  revise  the  work  accomplished  under 
Item  5,  as  related  to  the  further  objective 
to  achieve  a comprehensive,  detailed  design 
to  meet  both  basic  data  (base  line  data)  and 
special  data  requirements. 

Item  13. 

Refine  and  revise  the  work  accomplished  under 
Item  6,  as  related  to  the  further  objective 
of  collecting  both  basic  and  special  data. 

Item  14. 

Refine  and  revise  the  work  accomplished  under 
Item  7,  as  related  to  the  analysis  of  both 
basic  and  special  data;  and  provide  findings 
and  recommendations. 

Phase  Three:  On-line/Turnkey  Evaluation 


Items  15 
thru  21. 

Revision  and  refinement  of  respective  prior 
items  in  relation  to  the  additional  objective 
of  an  on-line  turnkey  evaluation  process. 

22 


RFP  6-35756 


STATEMENT  OF  WORK 

(with  illustrative  sub-items  for  Phase  One) 

Note:  The  items  listed  below  provide  a categorized 

summary  of  the  services  and  materials  required 
on  this  program. 

Phase  One  Planning  and  Design , and  Preliminary  Evaluation 

Item  1.  Plan,  organize,  staff,  direct,  and  control,  including 

both  schedule  and  costs. 

a.  Provide  the  necessary  services  and  materials  for 
management  of  the  overall  program,  including  a 
management  plan. 

b.  Prepare  a preliminary  detailed  schedule,  and 
provide  progressive  analysis  and  rescheduling, 
including  proposed  schedule  for  Phase  Two. 

c.  Prepare  periodic  cost  analyses  and  projections, 
including  proposed  costs  for  Phase  Two. 
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Item  2.  Provide  reports,  liaison,  and  services  for  purposes 
of  review,  coordination,  and  approvals  and  accept- 
ances, including  the  following: 

Provide  services  and  materials  necessary  for  liaison 
with  ETIP,  PAA,  and  relevant  other  individuals  and 
organizations . 

a.  Periodic  and  special  formal  and  informal  manage- 
ment progress  reports,  final  report 


a ( 1 ) . 

Submit  program  management  plan  at  or  before 
completion  of  3d  month,  and  report  modifica- 
tions as  required. 

a (2)  . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

b.  Periodic  and  final  scheduling  reports,  including 
projected  schedule  for  the  next  phase 


b ( 1 ) . 

Submit  projected  Phase  One  schedule  at  or 
before  completion  of  3rd  month,  and  report 
modifications  as  required. 

b (2 ) . 

Submit  projected  Phase  Two  schedule  at  or 
before  completion  of  9th  month. 

b ( 3 ) . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

c.  Periodic  and  final  cost  reports,  including  projected 
costs  for  the  next  phase. 


c(l)  . 

Submit  projected  Phase  One  costs  at  or  before 
completion  of  3rd  month,  and  report  modifi- 
cations as  required. 

c (2)  . 

Submit  final  report  at  or  before  completion 
of  15th  month. 

d.  Periodic  and  special  formal  and  informal  technical 
progress  reports,  final  report,  including  proposed 
program  for  the  next  phase.  (Note:  The  substantive 
work  to  be  reported  here  is  to  be  performed  under 
Items  3 through  7.) 
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Submit  reports,  including  formal  reports  for  the  items  in  the 
following  table  on  or  before  completion  of  the  month (s) 
specified : 


3a 

3rd  and  6th 

6a 

6th 

b 

3rd 

b 

- 

c 

6th 

c 

6th 

d 

9th 

d 

- 

4a 

3rd  and  6th 

7a 

6 th 

b 

6th  and  12th 

b 

12th 

5a 

3rd  and  6th 

c 

12th 

b 

6th 

d 

15th 

c 

6th 

e 

15th 

d 

12th 
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Item  3.  Review  and  structure  the  proposed  objectives  in 
terms  of  the  experimental  setting  and  related 
background,  and  establish  an  overall  plan  and 
set  of  priorities. 

a.  Review,  through  documents  and  interviews,  the 
objectives  of  ETIP,  PAA,  and  other  relevant 
individuals  and  organizations,  including 
professional  standards  of  evaluation,  and  prepare 
a description  and  supporting  analysis  suitable 
for  providing  a basis  for  planning  and  setting 

of  priorities  for  Phase  One. 

b.  Based  on  the  above,  identify,  and  prepare  pre- 
liminary recommendations  and  supporting  detail 
requirements  for,  those  specific  evaluations 
which  should  be  accomplished  during  Phase  One, 
including  both  exploratory  (and/or  descriptive) 
evaluations  and  a priori  proposition  testing 
evaluations . 

c.  Based  on  the  above,  prepare  preliminary  statement 
of  requirements  for  the  pilot  evaluation  system 
in  Phase  One. 

d.  Based  upon  the  above  and  the  other  activities 
in  Phase  One,  prepare  a preliminary  statement 

of  the  requirements  for  the  prototype  evaluation 
system  in  Phase  Two,  together  with  supporting 
analysis . 
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Item  4.  Review  and  structure  the  proposed  set  of  experiments, 
and  related  background  in  terms  of  the  proposed 
objectives,  and  provide  a reference  base. 

a.  Review,  through  documents  and  interviews,  the 
specific  procurement  experiments  (past,  present, 
and  proposed) , and  the  related  programs  and 
organizations*,  and  prepare  a description  and 
supporting  analysis  suitable  for  planning 

the  activities  of  Phase  One  in  conjunction  with 
the  work  in  Item  3a. 

* (AGENCY  IMPACT  ONLY)  With  primary  emphasis 
on  PAA,  and  not  on  industry  sector. 

(COMMERCIAL  IMPACT  ONLY)  With  primary  emphasis 
on  industry  sector,  and  not  on  PAA. 

b.  Prepare  a preliminary  framework  model  which 
provides  a base  for  synthesizing  objectives, 
organization,  programs,  and  the  evaluation 
process  (system),  with  supporting  analysis. 
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Item  5.  Develop  the  detailed  evaluation  requirements  (problem 
definition  and  experimental  or  study  design) , in 
relation  to  collection  and  analysis. 

a.  Prepare  a preliminary  design  of  both  exploratory 
(and/or  descriptive)  and  a priori  evaluations  to 
be  conducted  during  Phase  One,  including  1)  state- 
ment of  problem  or  hypothesis,  2)  supporting 
theories,  and  3)  definitions  of  variables  and 
parameters . 

b.  Based  on  the  above,  prepare  a detailed  design  of 
each  evaluation,  with  special  emphasis  on  specific 
procurement  experiments,  including  1)  choice  of 
experimental  design,  2)  (where  appropriate) 
sampling  strategies,  and  3)  data  collection  and 
analysis  methods. 

c.  Prepare  a preliminary  design  of  the  pilot  evalua- 
tion system  in  Phase  One,  including  data  collection 
and  analysis. 

d.  Based  on  the  above,  prepare  a preliminary  design  of 
the  prototype  evaluation  system  in  Phase  Two, 
including  both  basic  and  special  processes. 
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Item  6.  Develop  the  detailed  data  collection  process,  in- 
cluding instruments,  identification  of,  and  access 
to,  sources,  and  managing  the  process;  and  collect 
the  data. 

a.  Design  and  develop  the  data  collection  process 
required  for  the  specific  Phase  One  evaluations. 

b.  Based  upon  the  work  in  Items  3,  4,  and  5,  collect 
additional  data  required  for  specific  exploratory 
and/or  descriptive)  evaluations. 

c.  Design  and  pilot  test  the  instruments  necessary  for 
specific  a priori  proposition  testing  evaluations. 

d.  collect  the  data  reauired  for  specific  a priori 
proposition  testing  evaluations. 
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Item  7.  Develop  the  detailed  data  analysis  process;  carry  out 

the  analyses?  present  findings;  present  recommendations. 

a.  Design  and  develop  the  data  analysis  process  re- 
quired for  the  specific  Phase  One  evaluations. 

b.  Carry  out  the  specific  Phase  One  exploratory  (and/or 
descriptive)  evaluations,  and  present  findings. 

c.  Carry  out  the  specific  Phase  One  a priori  proposition 
testing  evaluations,  and  present  findings. 

d.  Analyse  the  activities  of  Phase  One  which  affect  the 
design  of  the  evaluation  system,  and  prepare 
recommendations . 

e.  Analyse  the  activities  of  Phase  One,  and  prepare 
recommendations  for  Phase  Two. 
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Item 

Item 

Item 

Item 

Item 

Item 

Item 


Phase  Two:  Prototype  Evaluation  Process  Testing 


8.  Same  as  Item  1. 

9.  Same  as  Item  2,  but  change  reference  to  Items  10 
through  14. 

10.  Refine  and  revise  the  work  accomplished  under 
Item  3,  as  related  to  the  further  objective  of 
designing  and  testing  a prototype  of  the  complete 
evaluation  process. 

11.  Refine  and  revise  the  work  accomplished  under  Item  4, 
related  to  the  further  objective  of  preparing  a 
formal  framework  within  which  the  objectives,  organi- 
zations, and  programs  may  be  related  to  the  evaluation 
process . 

12.  Refine  and  revise  the  work  accomplished  under  Item  5, 
as  related  to  the  further  objective  to  achieve  a 
comprehensive  detailed  design  to  meet  both  basic 
data  (base  line  data)  and  special  data  requirements. 

13.  Refine  and  revise  the  work  accomplished  under  Item  6, 
as  related  to  the  further  objective  of  collecting  both 
basic  and  special  data. 

14.  Refine  and  revise  the  work  accomplished  under  Item  7, 
as  related  to  the  analysis  of  both  basic  and  special 
data;  and  provide  findings  and  recommendations. 


31 


RFP  6-35756 


Phase  Three;  On-Line/Turn-Key  Evaluation 


Items  15  through  21.  Revision  and  refinement  of  respective 

prior  items,  in  relation  to  the  addi- 
tional objective  of  an  on-line/turn- 
key evaluation  process. 
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PROPOSED  SCHEDULE  - OVERALL 
(Agency  Impact  & Commercial  Impact) 


ITEM  DESCRIPTION 

PHASE  ONE 

1.  PROGRAM  MANAGEMENT 

2.  REPORTS  & REVIEWS 

3.  OBJECTIVES 

4.  BACKGROUND 

5.  EVALUATION  DESIGN 

6.  DATA  COLLECTION 

7.  DATA  ANALYSIS 


PHASE  TWO 

8.  PROGRAM  MANAGEMENT 

9.  REPORTS  & REVIEWS 

10.  OBJECTIVES 

11.  BACKGROUND 

12.  EVALUATION  DESIGN 

13.  DATA  COLLECTION 

14.  DATA  ANALYSIS 


PHASE  THREE 


15.  PROGRAM  MANAGEMENT 

16.  REPORTS  & REVIEWS 

17.  OBJECTIVES 

18.  BACKGROUND 

19.  EVALUATION  DESIGN 

20.  DATA  COLLECTION 

21.  DATA  ANALYSIS 

NOTE:  f (FORMAL)  PROGRESS  REPORT 
■ FINAL  REPORT 


. MINOR  EFFORT 

COORDINATION  OR  REPORT  WRITING 

ONLY 
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Tasks,  plus 
selected 
sub-tasks 


PROPOSED  SCHEDULE  - PHASE  ONE 
(Agency  impact  & Commercial  impact) 


Months 


1.  PROGRAM  MANAGEMENT 
Phase  Two  cost  proposal 

2.  REPORTS  & REVIEWS 
Design  of  reporting  system 

3.  OBJECTIVES 
Preliminary  analysis 

Identification  of  specific  Phase  One  evaluations 
Preliminary  evaluation  system  objectives 
Evaluation  system  objectives 

4.  BACKGROUND 
Preliminary  analysis 

Analysis  of  relevant  organizations 
Preliminary  framework 

5.  EVALUATION  DESIGN 
Analysis  of  Phase  One  evaluations 

Design  of  Phase  One  evaluations 
(including  sampling  strategies) 

Preliminary  design  of  evaluation  system 
(including  collection  & analysis) 

Design  of  evaluation  system 
(including  basic  &.  special) 

6.  DATA  COLLECTION 
General  data  collection 
Instrument  design  & pilot  test 
Data  on  specific  experiments 

7.  DATA  ANALYSIS 
Exploratory  analysis  & findings 

A priori  analysis  & findings 
Analysis  of  evaluation  system 
Recommendations 


1011 1 


12 


1 3 1 1 4 1 15  j 


1 

1 
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PROPOSAL  EVALUATION  FACTORS 


I . Management 

This  factor  is  a measure  of  the  background 
and  overall  management  capability  of  the  bidder. 

1.1  Evaluation  Management  Experience 

This  sub-factor  measures  the  bidder's  experience 
and/or  capability  of  the  personnel  he  will 
use  on  this  program. 

1.2  Importance  of  Program  to  the  Bidder 

This  sub-factor  is  a measure  of  the  degree  to 
which  the  proposal  represents  the  approval  and 
direction  of  the  company  rather  than  the  efforts 
of  professional  proposal  writers,  and  also  is  a 
measure  of  the  warranty  offered  by  the  company 
to  back  the  program  with  the  necessary  resources. 

1.3  Program  Management 

This  sub-factor  is  the  measure  of  the  bidder's  own 
proposed  method  for  planning,  organizing,  and 
controlling  the  program. 

1.4  Schedule  and  Cost  Control 

This  factor  measures  the  bidder's  ability  to 
predict  the  time  and  cost  requirements  of  his 
efforts,  to  plan  the  interrelation  of  the  several 
parts,  and  to  exercise  the  necessary  control. 
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II.  1 


II . 2 


II . 3 


II.  4 


Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  government  is 
able  to  effectively  and  progressively  review 
a very  large  scale  effort  with  a minimum  of 
people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  with  those  of 
other  related  individuals  and  organizations. 

Reports  and  Liaison  with  the  Experimental 
Technology  Incentives  Program  (ETIP) . 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  the 
Experimental  Technology  Incentives  Program 
(ETIP) . 

Reports  and  Liaison  with  the  Program  Administrative 
Agency  (PAA) 

This  sub-factor  measures  that  part  of  the  overall  factor 
which  relates  to  the  program  administrative  agency 
(and,  where  appropriate,  its  subelements  or  related 
organizations) . 

Liaison  with  Others 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  individuals 
and  organizations  other  than  those  described 
above . 

Approvals  and  Acceptances 

This  sub-factor  measures  the  bidder's  understanding 
and  acceptance  of  the  necessity  for  progressive 
and  substantial  approval  and  acceptance  during 
each  phase,  as  well  as  at  the  end  of  each  phase. 
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III . 


III.  1 


III.  2 


III.  3 


III.  4 


III.  5 


Objectives 

/ 

This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  objectives  of  this  procurement 
(and  related  programs) . 

ETIP  Evaluation  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  specific  objectives  of  this  procurement. 

Program  Administrative  Agency  (PAA)  Objectives 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  specific  objectives  of  the 
program (s)  and/or  experiment (s)  that  are  to  be 
evaluated . 

Other  Relevant  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  relevant  and  related  objectives,  goals, 
criteria,  standards,  and  the  like,  including  the 
state  of  the  art  in  administrative  experimentation 
(and/or  evaluation) . 

Phase  Two  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  incorporation  of  the  results  of 
Phase  One  in  the  restated  objectives  for  Phase  Two. 

Phase  Three  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  reformulation  of  objectives  in 
Phase  Three. 
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IV. 


IV.  1 


IV.  2 


IV.  3 


IV.  4 


IV.  5 


Background  Information 

This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  background  and  characteristics 
of  this  procurement. 

ETIP 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  ETIP  and  the  specific  program (s)  and/or 
experiment (s) , in  terms  of  past  history,  present 
status  and  plans  for  the  future. 

Program  Administrative  Agency  (PAA) 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  program  administrative 
agency  (and,  where  appropriate,  its  subelements 
or  related  organizations),  its  organization, 
policies  and  procedures. 

Other  Relevant  Background 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  programs,  organization,  policies 
and  procedures  of  other  relevant  organizations. 

Phase  Two  Background 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  prepare  a comprehensive  framework 
or  model  of  the  program(s)  and/or  experiment (s) 
and  the  evaluation  process. 

Phase  Three  Background 

This  sub-factor  is  a measure  of  the  bidder's  aware- 
ness of  possible  further  refinements  in  the  model 
necessary  for  the  continuation  of  the  evaluation 
process . 
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V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder's  capabilities 
to  carry  out  the  evaluation  design  requirements. 

V.l  Exploratory  (Descriptive)  and  A Priori  Proposition 

Testing  Hypotheses,  Variables  and  Parameters 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  define  the  detailed  evaluation  objectives. 

V.2  Experimental  and/or  Study  Designs 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  define  the  overall  experimental 
and/or  study  design. 

V.3  Data  Collection  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  design  the  overall  data  collection  process,  including 
sources  of  information,  sampling  strategies,  timing, 
etc . 

V.4  Data  Analysis  Design 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  design  the  overall  data  analysis  process, 
including  processing  and  analysis,  and  presenting 
results  and  recommendations. 

V.5  Phase  Two  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  redefine  the  detailed  experimental  and/or  study 
design  to  reflect  the  results  of  Phase  One,  the 
several  objectives,  and  model  of  the  process  in  a 
comprehensive  detailed  design  to  meet  both  basic 
data  (base  line  data)  and  special  data  requirements. 

V.6  Phase  Three  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to 
further  redefine  the  above,  to  provide  for  a detailed 
design,  and  necessary  supporting  materials,  which  would 
allow  an  on-going  evaluation  process. 
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VI. 


VI.  1 


VI.  2 


VI.  3 


VI.  4 


VI.  5 


Data  Collection  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
acquisition  of  data  on  the  variables  and  parameters 
of  interest. 

Instrument  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  develop  (and  test)  the  various  questionnaires, 
protocols,  schedules,  and  the  like  required  for  data 
collection. 

Information  Sources 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  identify  the  various  information 
sources,  and  to  gain  access. 

Data  Collection 

This  sub-factor  is  measure  of  the  bidder's  capability 
to  organize,  staff,  train,  direct  and  control  the 
personnel  who  obtain  the  data  by  survey,  interview, 
observation,  or  use  of  records. 

Phase  Two  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  requirements  of  the  Phase  Two 
collection  of  both  basic  (base  line)  and  special 
data . 

Phase  Three  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  plan  against  the  requirement  to  design  an 
ongoing  collection  process. 
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VII . 


VII . 1 


VII . 2 


VII . 3 


VII . 4 


Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
analysis  of  the  data  and  to  present  the  results  and 
recommendations . 


Analysis 

This  sub-factor  is  the  measure  of  the  bidder's  ability 
to  identify  and  apply  the  appropriate  analytical 
(including  statistical)  techniques  to  the  data 
collected . 

Exploratory  (and/or  Descriptive)  Findings 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  exploratory 
findings . 

A Priori  Proposition  Testing  Findings. 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  a priori 
f indings . 

Recommendations 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  relationship  between  the 
results  of  Phase  One  and  the  work  in  Phase  Two, 
in  terms  of  additional,  revised,  or  deleted 
hypotheses,  changes  in  objectives  or  background 
information,  and  the  relative  feasibility  and/or 
usefulness  of  alternative  designs  for  evaluation. 
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2/1/76 

PROPOSAL  EVALUATION  FACTORS 
(With  Illustrative  Questions) 


Note:  These  factors  and  sub-factors  apply  to  both  agency 

and  commercial  impact  proposals.  Where  specific 
illustrative  questions  apply  primarily  to  either  agency 
or  commercial  impact  only,  they  are  so  identified. 

I.  Management 

This  factor  is  a measure  of  the  background 
and  overall  management  capability  of  the  bidder. 

1.1  Evaluation  Management  Experience 

This  sub-factor  measures  the  bidder's  experience 
and/or  capability  of  the  personnel  he  will 
use  on  this  program. 


1.1.1  Discuss,  with  specific  examples,  recent  experience  in  the 
management  of  field  experiments,  projects,  and/or  programs. 
Indicate  size  and  complexity  of  program,  extent  of  own  responsi- 
bility for  design  of  the  program  (or  experiment)  and  the  evalua- 
tion. Indicate  by  tabulation  of  cross-references  other  sub- 
sections of  this  proposal  where  you  have  referred  to  any  program 
discussed  here.  Outline  the  original  contracted-f or  schedules 
and  costs  and  requirements,  and  explain  changes.  If  you  have 
had  any  "bad"  experience  here,  discuss  the  reasons  and,  if 
appropriate,  indicate  how  you  would  prevent  them  from  occurring 
on  this  program.  If  appropriate,  you  should  provide  the 
names,  addresses,  and  telephone  numbers  of  cognizant  COTR's. 

1.1.2  If  you  do  not  have  the  kind  of  experience  discussed  above, 
discuss  your  basis  for  believing  that  you  can  meet  the  require- 
ments of  this  program.  Discuss  in  terms  of  the  personnel  you 
intend  to  use. 

1.1.3  How  will  you  combine  and  coordinate  the  efforts  of 
staff  members  who  are  familiar  with  the  procurement  process, 
the  program  administrative  agency,  or  the  commercial  sector 
with  those  staff  members  who  have  the  technical  skills  in 
evaluation? 
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1.2  Importance  of  Program  to  the  Bidder 

This  sub-factor  is  a measure  of  the  degree  to 
which  the  proposal  represents  the  approval  and 
direction  of  the  company  rather  than  the  efforts 
of  professional  proposal  writers,  and  also  is  a 
measure  of  the  warranty  offered  by  the  company 
to  back  the  program  with  the  necessary  resources. 


1.2.1  What  is  the  relative  size  of  this  program  with  respect 
to  the  overall  sales  of  the  company?  If  your  company  has 
significant  decentralization  or  there  are  other  factors  which 
should  be  presented  for  clarity,  please  furnish  this  back- 
ground. If  the  proportionate  size  is  relatively  large,  why 
do  you  think  that  the  problems  associated  with  the  "single 
product"  will  not  arise?  If,  on  the  other  hand,  the  propor- 
tionate part  is  relatively  small,  what  assurance  is  there 
that  this  program  will  not  be  downrated  by  others  after  the 
contract  is  signed? 

1.2.2  Discuss,  if  appropriate,  how  this  program  fits  in  with 
the  long  range  planning  of  your  company.  What  is  its  signifi- 
cance to  your  future  position?  If  you  envisage  an  increase  in 
programs  of  this  type,  what  assurance  is  there  that  key  personnel 
on  this  program  will  not  be  withdrawn  for  newer  programs? 

1.2.3  It  should  be  noted  that  the  extent  and  nature  of  your 
effort  in  proposing  will  be  considered  in  the  evaluation  of 
this  sub-factor. 
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1.3  Program  Management 

This  sub-factor  is  the  measure  of  the  bidder's  own 
proposed  method  for  planning,  organizing,  and 
controlling  the  program. 


1.3.1  It  is  proposed  that  the  planning  and  carrying  out 

of  the  preparation  of  this  proposal  provides  an  opportunity 
for  the  bidder  to  demonstrate  his  capability  in  planning  a 
complex  effort.  Please  describe  briefly  your  process  and, 
particularly,  the  extent  to  which  it  is  a "pilot"  of  the 
process  you  will  use  in  managing  this  program. 

1.3.2  Detail  your  proposed  organization  by  time  phases, 
including  buildup,  division  of  responsibility;  generally 
describe  its  operation,  how  it  is  organized,  and  special 
features  not  self-explanatory.  Can  the  effectiveness  of 
your  proposed  organization  be  projected  from  its  performance 
in  putting  together  this  proposal?  If  the  organization  you 
propose  is  essentially  "new,"  either  in  form  or  in  the 
persons  assigned,  comment  on  the  extent  to  which  you  have 
been  able  to  "test"  it  out  in  preparing  this  proposal. 


44 


RFP  6-35756 


1.4  Schedule  and  Cost  Control 

This  factor  measures  the  bidder's  ability  to 
predict  the  time  and  cost  requirements  of  his 
efforts,  to  plan  the  interrelation  of  the  several 
parts,  and  to  exercise  the  necessary  control. 

1.4.1  A detailed,  thorough  schedule  provides  not  only  a 
comprehensive  plan  but  also  a powerful  method  of  controlling 
and  reviewing  the  progress  of  the  program.  The  proposed 
schedule  does  not  include  a detailed  breakdown  of  all  of 
the  interrelated  areas  but  does  furnish  a broad  guide  for 
planning  purposes.  In  general,  do  you  think  the  schedule 

is  realistic?  Is  it  too  tight?  What  additional  assumptions 
are  required?  What  conditions  or  events  will  affect  the 
schedule? 

1.4.2  Discuss  the  method  of  preparation  of  the  schedule  you 
will  use,  how  you  will  update  it,  how  you  will  use  it. 

1.4.3  Prepare  a supporting  analysis  and  summary  of  any  areas 
which  you  believe  reflect  a significant  relative  cost  advantage 
or  disadvantage  on  your  part.  Indicate  to  what  extent  your 
in-house  or  subcontracting  plans  will  take  advantage  of 
in-being  capabilities  and  at  what  potential  savings.  To 

what  extent  have  provisions  for  contingencies  been  made? 

1.4.4  Discuss  the  form  and  timing  of  your  detailed  cost 
proposal  for  the  major  field  data  gathering  activity  in 
the  second  and  third  phases. 

1.4.5  While  the  overall  schedule  reflects  the  requirement 
to  design  an  on-going  evaluation  system,  there  is  an 
important  requirement  to  provide  during  Phase  One  (and  each 
subsequent  phase)  specific  outputs  in  the  form  of  evaluations 
of  selected  specific  experiments.  How  will  you  assure  this? 

1.4.6  The  timing  of  the  experiments  varies  across  the  several 
PAA's.  What  advantages  or  disadvantages  do  you  see  in  this? 

1.4.7  It  is  anticipated  that  another  ("agency  impact") 
contractor  will  be  primarily  responsible  for  evaluating  the 
intervention  and  proximal  impacts  of  the  several  experiments; 
it  is  also  anticipated  that  some,  if  not  all,  commercial 
impacts  will  substantially  lag  the  experiments.  What  advantages 
or  disadvantages  do  you  see  in  this?  (COMMERCIAL  IMPACT  ONLY) 
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II.  Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  government  is 
able  to  effectively  and  progressively  review 
a very  large  scale  effort  with  a minimum  of 
people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  with  those  of 
other  related  individuals  and  organizations. 

II. 1 Reports  and  Liaison  with  the  Experimental 

Technology  Incentives  Program  (ETIP) 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  the 
Experimental  Technology  Incentives  Program 
(ETIP) . 


II.  1.1  Discuss  generally  the  extent  and  nature  of  liaison 
with  both  the  Contracting  Officer  (CO)  and  the  Contracting 
Officer's  Technical  Representative  (COTR)  that  you  envision 
in  this  program.  Discuss  generally  the  information  needs  of 
both  government  and  contractor  personnel;  who  will  establish 
what  the  needs  are  and  how  they  will  be  satisfied? 

II. 1.2  It  would  be  useful  if  the  reporting  system  were 
designed  in  a way  that  facilitated  an  ongoing  process 
evaluation  of  a)  the  experiments  conducted  by  the  program 
administrative  agency,  and  b)  the  evaluation  effort  of  the 
contractor  himself.  The  data  provided  by  such  a system  could 
serve  as  the  basis  for  a descriptive  case  history  of  the 
experiments  and  their  evaluation.  Discuss  generally  how 
feasible  this  proposal  is,  and  how  it  might  be  accomplished. 
You  may  wish  to  relate  it  to  your  discussion  in  IV. 1.2,  and 
comparable  discussions. 
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II. 2 Reports  and  Liaison  with  the  Program  Administrative 

Agency  (PAA) 

This  sub-factor  measures  that  part  of  the  overall  factor 
which  relates  to  the  program  administrative  agency 
(and,  where  appropriate,  its  subelements  or  related 
organizations) . 

11. 2.1  Discuss  generally  the  extent  and  nature  of  liaison  with 
both  management  personnel  and  technical  or  specialized  staff. 

How  will  you  reconcile  the  desirability  of  direct,  informal  com- 
munication with  the  necessity,  in  at  least  some  cases,  for 
observing  formal  clearance  and  approval  "through  channels"? 

11. 2. 2 The  evaluation  effort  may  sometimes  be  perceived  as 
threatening  by  various  personnel  in  the  program  administra- 
tive agency.  Discuss  how  the  reporting  and  liaison  arrange- 
ments can  be  developed  to  reduce  the  likelihood  of  this. 

11. 2. 3 Discuss  the  differences,  if  any,  in  how  you  will  relate 
to  the  several  PAA.  To  what  extent  will  these  be  separately 
identifiable  efforts? 

11. 2. 4 It  is  anticipated  that  another  ("agency  impact") 
contractor  will  be  primarily  responsible  for  coordination 
with  the  several  PAA.  What  do  you  think  of  this,  and  how 
will  you  deal  with  it?  (COMMERCIAL  IMPACT  ONLY) 


47 


RFP  6-35756 


II.  3 Liaison  with  Others 

This  sub-factor  measures  that  part  of  the 
overall  factor  which  relates  to  individuals 
and  organizations  other  than  those  described 
above . 

11. 3.1  It  is  anticipated  that  there  will  be  related  programs 
and  program  evaluations  being  carried  out  by  the  government 
agencies  either  in-house  or  through  the  use  of  an  outside 
consultant  or  contractor.  Discuss  the  nature  and  form  of 
the  relationship  you  see  with  them.  How  will  you  ensure 
that  the  necessary  coordination,  avoidance  of  duplication, 
overlap,  or  gaps,  etc.,  is  accomplished? 

11. 3.2  To  some  extent,  it  is  expected  that  individual  companies 
(e.g.,  manufacturers  of  products  which  are  the  subject  of  pro- 
gram experiments),  industry  associations,  and  others  may  desire 
or  require  a liaison  relationship.  Comment. 

11. 3. 3 Reporting  and  liaison  relationships  may  arise  with 
individuals  and  organizations  with  professional  interests 
(e.g.,  professional  societies,  conferences,  professional 
journals,  consumer  or  public  interest  groups)  in  the  pro- 
gram or  program  evaluation.  Discuss. 


11. 3.4  It  is  anticipated  that  another  ("commercial  impact") 
contractor  will  be  primarily  responsible  for  coordination  with 
individual  companies,  etc.  Comment.  (AGENCY  IMPACT  ONLY) 

11. 3.5  It  is  anticipated  that  the  bidder  will  be  primarily 
responsible  for  coordination  with  individual  companies,  etc. 
Comment.  (COMMERCIAL  IMPACT  ONLY) 
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II. 4 Approvals  and  Acceptances 

This  sub-factor  measures  the  bidder's  understanding 
and  acceptance  of  the  necessity  for  progressive 
and  substantial  approval  and  acceptance  during 
each  phase,  as  well  as  at  the  end  of  each  phase. 

II. 4.1  It  is  intended  that  the  formal  approval  and  acceptance 
of  the  work  of  the  contractor  for  each  phase  will  be  based, 
primarily,  upon  the  several  reports  submitted  during  and 
at  the  end  of  the  phase,  with  final  approval  at  the  end  of 
the  phase.  It  is  intended,  however,  that  this  final  approval 
and  acceptance  will  be  largely  a formal  confirmation  of 
already  established  acceptable  performance  and/or  results. 
This  is  required  not  only  to  overcome  the  substantial  delays 
of  tandem  or  sequential  review,  but  also  to  avoid  rework, 
redesign  and  argument  under  pressure  of  time  and  after  the 
fact.  You  should  discuss  your  understanding  of  this,  the 
advantages  and  disadvantages,  and  your  willingness  to  accept 
the  contractual  and  schedule  implications,  that  is,  that  this 
means  that  the  delivery  dates  are  dates  of  delivery  of 
approved  items,  not  merely  dates  for  submission  of  items 
with  an  open  time  term. 
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III.  Objectives 


This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  objectives  of  this  procurement 
(and  related  programs). 

111.1  ETIP  Evaluation  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  specific  objectives  of  this  procurement. 

111. 1.1  The  specific  objectives  for  this  procurement,  and 
related  objectives,  including  the  overall  objectives  of 
ETIP,  appear  in  various  forms  in  the  several  parts  of  this 
bid  set,  and  in  other  sources,  and  reflect  a variety  of 
authors,  circumstances  and  purposes.  It  is  proposed  that 
it  is  neither  necessary  nor  desirable  for  ETIP  to  prepare 

a single,  integrated,  fully  rationalized,  and  "authoritative" 
statement  of  the  objectives  for  the  guidance  of  the  bidders 
(or  the  successful  bidder) . Comment  on  why  you  believe  this 
is  (or  is  not)  justified? 


III. 1.2  It  is  expected  that  the  various  stated  objectives 
may  not  appear  (or  be)  consistent;  how  will  you  handle  this? 
For  example,  a high  credibility  (a  priori  proposition  testing) 
evaluation  of  a specific  experiment  is  desired  but  it  is  not 
possible  to  achieve  this  (with  reasonable  time  and  money) . 

For  example,  a complete  and  candid  description  of  how  a 
particular  experiment  was  carried  out  may  reflect  gross 
incompetence  or  dereliction  on  the  part  of  specific  ETIP 
(or  program  administrative  agency)  personnel.  For  example, 
the  results  of  a particular  evaluation  may  disclose  a 
systematic  defect  in  the  past  performance  of  the  agency 
(or  a contractor) . 


111. 1.3  It  is  anticipated  that,  for  a particular  experiment, 
there  may  be  as  many  as  four  parallel  objectives:  a)  to 
describe  what  happened,  what  went  well,  what  went  wrong; 

b)  to  determine  whether  the  result  was  "caused"  by  the 
experimental  manipulation;  c)  to  demonstrate  the  ability 
to  "evaluate"  scientifically  with  credible  results;  d)  to 
claim  credit  for  the  experiment  and/or  the  results.  How 
does  one  determine  the  relative  importance  of  these  several 
objectives  for  a specific  experiment  over  the  set  of  all 
comparable  experiments? 

111. 1.4  It  is  anticipated  that  ETIP  evaluation  objectives  or 
priorities  may  shift  during  the  course  of  the  evaluation 
effort.  How  does  one  ensure  that  he  will  be  sensitive  to 
such  shifts  and  how  does  one  deal  with  such  shifts?  How  would 
you  go  about  improving  the  clarity  of  ETIP's  evaluation 
objectives? 
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III. 2 Program  Administrative  Agency  (PAA)  Objectives 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  specific  objectives  of  the 
program(s)  and/or  experiment (s ) that  are  to  be 
evaluated . 

III.  2.1  Questions  comparable  to  those  suggested  under  III.l 
may  appear  here.  Comment.  What  special  problems,  if  any, 
do  you  anticipate  because  of  the  multiple  PAA? 

III.  2. 2 How  will  you  reconcile  conflicts  between  ETIP  and 
the  program  administrative  agency  objectives  for  a particu- 
lar experiment?  PAA  objectives  for  the  experiment  and 
other  PAA  objectives?  What  kinds  of  problems  will  come 
up?  How  do  you  plan  to  deal  with  them? 

III.  2. 3 It  is  anticipated  that,  during  the  period  that  the 
work  on  this  contract  is  accomplished,  there  may  be 
changes  in  policy  or  procedures  or  of  key  personnel  of  the 
program  administrative  agency.  How  will  you  deal  with  this? 
For  example,  the  "results"  of  a particular  experiment  may  be 
sufficiently  "obvious"  to  the  agency  to  no  longer  make 
necessary  any  further  evaluation.  Will  a continuation  of 
the  evaluation  be  undertaken  under  any  circumstances?  And, 
if  so,  how  would  you  justify  it  and/or  obtain  agency  co- 
operation? 

III.  2. 4 Suppose  the  agency  adds  new  objectives  or  objectives 
not  presently  contemplated  are  uncovered  which  represent 
potentially  significant  changes  in  the  direction  or  scope 
of  the  present  evaluation  program?  How  will  you  react  to 
this? 

III.  2. 5 It  is  anticipated  that  another  ("agency  impact") 
contractor  will  be  primarily  responsible  for  examining  the 
objectives  of  the  PAA.  Comment.  (COMMERCIAL  IMPACT  ONLY) 
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III. 3  Other  Relevant  Objectives' 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  relevant  and  related  objectives,  goals, 
criteria,  standards,  and  the  like,  including  the 
state  of  the  art  in  administrative  experimentation 
(and/or  evaluation) . 

111. 3.1  It  is  proposed  that  the  objectives  of  ETIP  and  PAA 
do  not  exist  in  isolation,  that  both  specific  and  general 
objectives  of  a wide  variety  of  other  individuals,  organiza- 
tions, and  larger  communities  will  necessarily  interact  with 
the  present  program.  For  example,  personal  and  career  object- 
ives of  key  individuals.  For  example,  political,  social,  and 
legal  objectives  of  the  federal  government.  For  example, 
objectives  expressed  or  implied  in  the  state-of-the-art  and 
professional  standards  in  experimental  and  evaluation  research. 
Of  what  significance  are  these,  and  how  do  you  expect  to  deal 
with  them? 

111. 3.2  Your  understanding  of  the  state-of-the-art, 
particularly  in  administrative  experimentation  and/or 
evaluation,  will  be  measured  by  your  discussion  of  relevant 
questions  raised  elsewhere.  If  you  consider  it  appropriate, 
you  may  discuss  this  here,  and  you  may  wish  to  cross-reference 
significant  material  which  appears  elsewhere. 

111. 3. 3 Are  there  relevant  professional  standards  in 
experimentation  that  you  must  observe  that  may  conflict 
with  ETIP  or  PAA  objectives?  How  will  you  handle  this? 

You  may  wish  to  relate  your  answer  to  II. 3. 

111. 3. 4 It  is  anticipated  that  another  ("commercial  impact") 
contractor  will  be  primarily  responsible  for  examining  the 
objectives  of  individual  companies  and  others  in  the 
commercial  sector.  Comment.  (AGENCY  IMPACT  ONLY) 

111. 3.5  It  is  anticipated  that  the  objectives  of  individual 
companies,  associations,  etc.,  may  significantly  affect  not 
only  the  experiments  and  their  evaluation  but  also  the 
activities  of  the  bidder.  Comment.  (COMMERCIAL  IMPACT  ONLY) 
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III. 4 Phase  Two  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 

to  anticipate  the  incorporation  of  the  results  of 

Phase  One  in  the  restated  objectives  for  Phase  Two. 

111. 4.1  It  is  proposed  that,  as  a result  of  the  work  in 

Phase  One,  a reasonable  description  and  integration  of  the 
several  objectives  will  have  been  realized,  and  that  this 
will  be  used  as  a basis  for  the  several  other  tasks  but 
also  as  a basis  for  refinement,  revision,  and  additions 
during  Phase  Two.  In  addition  to  this  continuing  interest 
in  the  objectives  introduced  in  Phase  One,  it  is  proposed 
that  a separate  objective  of  Phase  Two  is  to  design  and 
test  a prototype  of  the  complete  evaluation  process 
necessary  to  meet  the  several  objectives.  It  is  expected 
that  this  may  include  at  least  two  separately  identifiable 
parts,  as  follows:  a basic  evaluation  process  to  include 

those  evaluation  activities  which  deal  with  key,  predictable, 
identifiable,  continuing,  common  objectives  for  which  the 
process  can  be  "institutionalized";  and  a special  evalua- 
tion process  to  include  those  evaluation  activities  which 

do  not  meet  the  above  requirements  because  of  specialized, 
one-time,  or  changing  objectives.  If  you  believe  your 
discussion  elsewhere  does  not  adequately  reflect  your  under- 
standing of  this,  you  may  comment  here. 

111. 4. 2 What  activities  during  Phase  One  will  be  expected 
to  contribute  to  the  refinement  of  Phase  Two  objectives? 

What  other  sources  may  be  expected  to  provide  input?  You 
may  wish  to  cross-reference  your  comments  under  IV. 4,  or 
elsewhere . 
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III.  5 Phase  Three  Objectives 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  reformulation  of  objectives  in 
Phase  Three. 

III. 5.1  In  addition  to  a continuation  of  the  concern  with  the 
objectives  of  the  previous  two  phases,  it  is  proposed  that 
the  objective  of  Phase  Three  will  be  to  provide  a firm  basis 
for  any  further  related  evaluation  efforts,  including  any  of 
the  following:  a)  a standby  or  "how  to"  capability  to  design 

and  carry  out  evaluations  for  comparable  future  experiments? 
b)  an  on-line,  turnkey  data  gathering  and  analysis  process 
which  might  be  implemented  by  the  PAA;  c)  an  assessment  of 
the  need  (or  lack  of  need)  for  other  specific  or  general 
future  evaluation  efforts.  How  realistic  do  you  think  this 
is,  and  to  what  extent  will  the  prior  phases  prepare  you  to 
successfully  define  and  then  meet  these  objectives? 
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Background  Information 

This  factor  is  a measure  of  the  bidder's  overall 
understanding  of  the  background  and  characteristics 
of  this  procurement. 

ETIP 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  ETIP  and  the  specific  program (s)  and/or 
experiment (s ) , in  terms  of  past  history,  present 
status  and  plans  for  the  future. 

IV. 1.1  Analogous  to  the  comments  in  III. 1.1,  it  is  not 
proposed  to  provide  a complete  and  comprehensive  description 
of  ETIP  and  the  specific  program(s)  and/or  experiment (s ) for 
the  guidance  of  the  bidders  (or  the  successful  bidder) . 

Discuss  the  adequacy  of  your  present  understanding  of  ETIP 
and  the  specific  programs;  if  you  believe  it  is  not  adequate, 
how  do  you  propose  to  correct  this  consistent  with  the 
requirements  of  Phase  One? 

IV. 1.2  It  is  assumed  that  a reasonably  complete  and  current 
description  of  at  least  the  specific  program (s)  and/or 
experiment (s ) will  be  desirable,  if  not  necessary,  during 
Phase  One  (and,  perhaps,  on  a continuing  basis)  for  use 
not  only  by  the  bidder  but  also  by  ETIP.  How  can  this  be 
accomplished?  What  forms  and  procedures  do  you  visualize 
will  be  used  to  provide  for  an  up-to-date  record?  How 
will  inputs  (revisions)  be  made?  How  will  outputs  be 
available? 

IV. 1.3  The  availability  of  such  an  information  base  (and 
system)  would  necessarily  increase  the  opportunities  to 
make  corrections  in  ongoing  activities  (experiments) 
and  may  suggest  other  improvements  and  changes  in  the 
program.  To  what  extent  do  you  see  it  as  your  responsibility 
to  facilitate  this  process?  What  effect  would  these  additional 
changes  have  on  your  evaluation  work? 

IV. 1.4  Discuss  your  understanding  of  the  overall  set  of 
programs  and/or  experiments  to  be  evaluated,  including  at 
least  the  following;  the  set  of  individual  product  procure- 
ments under  life  cycle  costing  (LCC)  both  planned  and 
implemented;  the  FSS  decision  to  adopt  LCC  as  a procurement 
practice;  the  several  workshops  and  the  training  program  as 
planned  and  implemented  in  all  FSS  offices  around  the  country, 
other  planned  or  implemented  procurement  experiments,  such 
as  value  incentive  contracting.  Can  these  be  described  or 
modeled  in  such  a form  as  to  disclose  their  interactions, 
or,  conversely,  can  specific  experiments  or  programs  be 
defined  in  such  a way  as  to  allow  separate  treatment? 


IV. 


IV.  1 


55 


RFP  6-35756 


IV. 2 Program  Administrative  Agency  (PAA) 

/ 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  program  administrative 
agency  (and,  where  appropriate,  its  subelements 
or  related  organizations),  its  organization, 
policies  and  procedures. 

IV. 2.1  Questions  comparable  to  those  suggested  under  IV. 1.1 
may  appear  here.  Comment.  What  special  problems,  if  any, 
do  you  anticipate  because  of  the  multiple  PAA? 

IV. 2. 2 As  a further  restatement  of  the  above,  if  you  do  not 
have  a present  acquaintance  with  the  program  administrative 
agency  (and  its  constituent  elements) , how  will  you  acquire 
the  necessary  background? 

IV. 2. 3 At  least  some  parts  of  the  organization,  policies, 
and  procedures  of  the  program  administrative  agency  will 
be  critically  related  to  the  present  program.  For  example, 
those  elements  directly  involved  in  the  experiments,  and 
procurement  policies  and  procedure.  Discuss  your  under- 
standing of  these  areas.  If  you  do  not  have  prior  experience, 
what  effect  will  this  have  on  your  capability? 

IV. 2. 4 It  is  anticipated  that  another  ("agency  impact") 
contractor  will  be  primarily  responsible  for  obtaining 
information  about  and  access  to  the  PAA.  How  will  you  deal 
with  this?  (COMMERCIAL  IMPACT  ONLY) 
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IV . 3 Other  Relevant  Background 

This  sub-factor  is  a measure  of  the  bidder's  under- 
standing of  the  programs,  organization,  policies 
and  procedures  of  other  relevant  organizations. 


IV. 3.1  It  is  anticipated  that  information  about  and  access 
to  organizations  other  than  ETIP  and  PAA  may  be  important 
to  this  program.  For  example:  other  government  agencies 
which  are  the  end  users  of  the  products  procured;  other 
government  agencies  with  legal,  financial,  policy,  or 
other  responsibilities  which  interact  with  the  program 
area;  specific  companies,  industry  segments,  and  industry 
associations;  consumer  organizations,  safety,  advertising, 
news  media,  and  the  like;  other  ETIP  or  PAA  program  or 
evaluation  contractors.  Discuss  your  understanding  of  the 
potential  effects  of  these  interactions  and  how  you  will 
deal  with  them. 

IV. 3.2  It  is  anticipated  that  another  ("commercial  impact") 
contractor  will  be  primarily  responsible  for  obtaining  information 
about  and  access  to  individual  companies  and  others  in  the 
commercial  sector.  Comment.  (AGENCY  IMPACT  ONLY) 

IV. 3. 3 It  is  anticipated  that  the  bidder  will  be  primarily 
responsible  for  obtaining  information  about  and  access  to 
individual  companies  and  others  in  the  commercial  sector. 

Comment.  (COMMERCIAL  IMPACT  ONLY) 
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IV. 4 Phase  Two  Background 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  prepare  a comprehensive  framework 
or  model  of  the  program (s)  and/or  experiment (s) 
and  the  evaluation  process. 


IV. 4.1  It  is  anticipated  that,  after  Phase  One,  the  bidder 
will  be  capable  of  preparing  a formal  framework  within 
which  the  objectives,  organizations,  and  programs  may  be 
related  to  the  evaluation  process.  Such  a framework  would 
not  only  integrate  the  background  from  Phase  One  but  provide 
a convenient  and  efficient  capability  for  revision,  additions, 
and  modifications  without  having  to  start  over  again.  Comment 
on  the  feasibility  of  this;  if  you  can,  suggest  how  this 
might  be  done  or  how  one  would  determine  whether  it  was 
successful  or  not?  You  may  wish  to  cross-reference  your 
comments  on  III. 4. 

IV. 4. 2 Will  the  distinction  between  agency  impact  and 
commercial  impact  require  significantly  different 
frameworks?  If  two  separate  frameworks  are  developed, 
to  what  extent  will  it  be  necessary  and/or  feasible  and/or 
desirable  to  relate  the  frameworks? 

IV. 4. 3 Discuss  the  same  question  with  respect  to  the 
distinction  among  the  several  PAA. 


58 


RFP  6-35756 


IV. 5 Phase  Three  Background 

This  sub-factor  is  a measure  of  the  bidder's  aware- 
ness of  possible  further  refinements  in  the  model 
necessary  for  the  continuation  of  the  evaluation 
process . 

IV. 5.1  It  is  proposed  that  the  model,  and  its  supporting 
descriptive  material  would  provide  a sufficient  basis  for 
a turnkey  operation  so  that  PAA  personnel  could,  with 
modest  specialized  assistance,  carry  out  on-going 
evaluations  for  comparable  programmatic  extensions. 

Comment  on  this. 

IV. 5. 2 It  is  anticipated  that  the  specific  PAA  would  be 
primarily  interested  in,  and,  be,  logically,  the 
appropriate  organization  to  manage,  the  on-going  "agency 
impact"  evaluation  system  (or  that  part  concerned  with 
the  specific  agency).  Comment.  (AGENCY  IMPACT  ONLY) 

IV. 5. 3 It  is  anticipated  that  significant  parts  of  the 
"commercial  impact"  evaluation  system  may  already  be  within 
the  present  capability  (or  future  plans)  of  related  organiza- 
tions (e.g..  Department  of  Commerce,  industry  associations, 
etc.).  What  advantages  or  disadvantages  do  you  see  in  this 
possibility,  and  how  will  you  react  to  it?  (COMMERCIAL 
IMPACT  ONLY) 
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V.  Evaluation  Design 

This  factor  is  a measure  of  the  bidder's  capabilities 
to  carry  out  the  evaluation  design  requirements. 

V . 1 Exploratory  (Descriptive)  and  A Priori  Proposition 

Testing  Hypotheses,  Variables  and  Parameters 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  define  the  detailed  evaluation  objectives. 


VI. 1.1  It  is  proposed  that  the  various  evaluation  design  require- 
ments will  include  problems  ranging  over  several  dimensions, 
and  combinations  of  dimensions,  including  the  following: 
a)  from  exploratory  and/or  descriptive  evaluations  (e.g., 
how  did  a specific  experiment  get  started;  what  happened; 
what  are  the  more  significant  secondary  effects)  to  a priori 
proposition  testing  (e.g.,  did  specific  intervention  X "cause" 
the  observed  result  Y) ; b)  from  very  specific,  identifiable 
experiments  (manipulations  and/or  effects)  to  general,  diffuse, 
overall  changes;  c)  from  post  hoc  evaluations  of  completed 
experiments  through  evaluations  of  ongoing  experiments,  to 
planning  and  design  of  future  experiments.  Comment  on  your 
understanding  of  this. 

V.1.2  At  least  initially,  the  bidder  will  be  called  upon  to 
accomplish  a preliminary  systems  design  or  study.  If  your 
evaluation  experience  has  been  primarily  in  response  to 
detailed  evaluation  requirements  provided  in  the  RFP  or 
RFQ , discuss  the  relevance  of  your  experience  or  capability 
to  the  requirements  of  this  program. 

V.1.3  Discuss  your  approach  to  the  detailed  definition  of 
a research  (or  experimental  or  evaluation)  problem;  comment 
on  theory  building,  definition  of  variables  and  parameters, 
etc.  If  considered  necessary,  distinguish  the  approach  to 
a priori  proposition  testing  from  that  used  in  exploratory 
and/or  descriptive  evaluation. 
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V.2  Experimental  and/or  Study  Designs 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  define  the  overall  experimental 
and/or  study  design. 


V.2.1  Discuss  the  problem (s)  of  and  approaches  to  experimental 
and/or  study  design.  Alternatively,  you  may  wish  to  demonstrate 
your  capability  by  reference  to  the  relevant  education  and 
experience  of  your  staff.  It  is  suggested  that  you  may  wish 
to  distinguish  among  the  several  kinds  of  designs  which  may 
be  appropriate  for  meeting  the  several  kinds  of  objectives 
which  are  the  outcome  of  the  process  discussed  in  V.l. 

V.2. 2 Discuss  the  problem (s)  of  and  approaches  to  identifying 
and  controlling  for  the  potential  effects  of  parameters. 

V.2. 3 it  is  proposed  that  the  bidder  can  design  an  overall 
evaluation  system  and,  in  parallel,  carry  out  evaluations  of 
specific  selected  experiments  during  Phase  One.  Discuss 
the  advantages  and  disadvantages  of  this.  Can  you  identify 
specific  experiments  which  can  be  evaluated  during  Phase 
One?  Can  you  identify  base-line  data  which  can,  or  should, 
be  collected  during  Phase  One?  which  can,  or  should,  be 
collected  during  later  phases? 

V.2. 4 To  what  extent  do  you  see  differences  among  the  several 
PAA  which  will  affect  the  design  of  the  evaluation  system? 

Will  differences  in  the  progress  in  initiating  experiments 
among  the  several  PAA  be  an  advantage  or  disadvantage? 

V.2. 5 It  is  anticipated  that  some  part  or  all  of  the  bidder's 
activity  in  this  area  will  be  dependent  upon  the  activities 
of  another  ("agency  impact")  contractor.  How  critical  is 
this  from  a time  point  of  view?  from  an  interface  point  of 
view?  Will  segmentation  (by  industry,  technology,  market) 
provide  advantages  or  disadvantages?  to  what  extent  will 
longitudinal  designs  be  important,  and  will  this  be  an 
advantage  or  disadvantage  (COMMERCIAL  IMPACT  ONLY) 
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V.3  Data  Collection  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  design  the  overall  data  collection  process,  including 
sources  of  information,  sampling  strategies,  timing, 
etc . 

V.3.1  Discuss  the  various  kinds  of  sources  or  methods  of 
collecting  information  (e.g.,  observations,  questionnaires, 
interviews,  records)  in  terms  of  your  expectations  for 
their  use  here,  and  their  advantages  and  disadvantages. 

V.3. 2 The  nature  and  extent  of  the  information  required  will, 
at  least  in  some  cases,  require  the  development  of  sampling 
strategies.  Describe,  by  reference  to  relevant  experience, 
if  appropriate,  your  capability  to  develop  sampling  strategies 
for  a priori  proposition  testing  evaluations.  You  may  wish 
to  discuss  the  approaches  used  for  exploratory  evaluation. 

You  may  wish  to  comment  on  some  of  the  sampling  strategies 
provided  as  illustrations  in  the  several  reference  documents. 

V.3. 3 Discuss  any  special  considerations  which  may  be  intro- 
duced by  the  differences  among  the  several  PAA. 

V.3.4  It  is  anticipated  that  the  evaluation  of  agency  impact 
and  commercial  impact  will  require  the  obtaining  of,  in 
some  cases,  not  only  the  same  data  but  also  separate 
additional  data  from  common  sources.  How  can  this  be 
coordinated  to  minimize  duplication  and,  particularly, 
the  burdening  of  sources? 
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V.4  Data  Analysis  Design 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  design  the  overall  data  analysis  process, 
including  processing  and  analysis,  and  presenting 
results  and  recommendations. 

V.4.1  It  is  proposed  that  this  part  of  the  process  should  be 
developed  concurrently  with  those  discussed  in  V.2  and  V.3 
instead  of  waiting  until  the  data  is  collected.  Comment  on 
the  desirability,  necessity,  and/or  feasibility  of  this. 

V.4. 2 Specific  individual  experiments  and  specific  exploratory 
studies  may  have  not  only  different,  but  also  severable 
analytical  requirements  (and  discussion  of  these  may  more 
appropriately  appear  in  VII) . It  is  expected,  however,  that 
there  may  be  some  degree  of  overlap  or  interrelation  among 
the  several  experiments  and  studies.  To  what  extent  would 
an  overview  of  the  overall  process  improve  the  ability  to 
perform  the  individual  analyses? 

V.4. 3 It  is  proposed  that  the  quantity  and  variety  of  the 
various  data  collected  (and  to  be  collected  in  the  future) 
present  an  opportunity  for  careful  planning  with  respect 
to  formatting  identification,  quality  control,  standardization, 
etc.  To  what  extent  do  you  think  this  is  necessary  and/or 
feasible?  To  what  extent  can  compatability  be  achieved  with 
related  existing  data  banks? 
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V.5  Phase  Two  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  redefine  the  detailed  experimental  and/or  study 
design  to  reflect  the  results  of  Phase  One,  the 
several  objectives,  and  model  of  the  process  in  a 
comprehensive  detailed  design  to  meet  both  basic 
data  (base  line  data)  and  special  data  requirements. 

V.5.1  Discuss,  or,  if  appropriate,  cross-reference  discussions 
elsewhere  of,  your  understanding  of  the  problems  outlined  in  V.l 
through  V.4  as  applicable  to  Phase  Two,  and  with  reference 
to  the  requirements  of  Phase  Two. 

V.5. 2 It  is  anticipated  that  the  bidder  will  provide  a 
comprehensive  and  detailed  design  of  an  evaluation  system 
which  will  provide  a capability  for  both  basic  and  special 
evaluation.  Discuss  your  present  understanding  of  the 
form  and/or  extent  of  the  evaluation  system.  Discuss  the 
advantages  and  disadvantages  of  prototype  testing  the  system 
by  evaluating  specific  procurement  or  other  experiments? 
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V.6  Phase  Three  Evaluation  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability  to 
further  redefine  the  above/  to  provide  for  a detailed 
design,  and  necessary  supporting  materials,  which  would 
allow  an  on-going  evaluation  process. 

V.6.1  Discuss  or,  if  appropriate,  cross-reference  discussions 
elsewhere  which  demonstrate  your  ability. 

V.6. 2 Discuss  your  understanding  of  what  will  be  necessary  to 
meet  turnkey  requirements.  What  consideration  of  agency  policies 
and  procedures  will  be  required?  How  will  the  necessary  skills 
and  experience  be  transferred?  How  will  previously  collected 
data  be  transferred?  What  conversion  of  instruments  and  software 
will  be  required?  What  will  be  the  effect  of  on-going  evaluations 
of  specific  experiments  on  the  transition  process?  and  vice  versa? 
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Data  Collection  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
acquisition  of  data  on  the  variables  and  parameters 
of  interest. 

Instrument  Design 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  develop  (and  test)  the  various  questionnaires, 
protocols,  schedules,  and  the  like  required  for  data 
collection . 

VI . 1 . 1 Discuss  (you  may  use  an  example  and/or  reference  to 
prior  experience)  the  problems  and  solutions  which  you  con- 
sider critical  (and/or  characteristic)  in  instrumentation 
for  an  a priori  proposition  testing  type  of  evaluation. 

VI. 2 Discuss  the  differences,  if  any,  in  the  approach  to 
descriptive  or  exploratory  evaluation. 

VI. 1.3  Under  what  circumstances,  and  for  what  purposes, 
will  you  conduct  pilot  tests?  Validation  tests? 

VI. 1.4  It  is  anticipated  that  some  variables  of  interest 
will  be  of  ongoing  importance  (as  key  variables  or  parameters 
common  to  a number  of  experiments,  or  as  key  exploratory 
or  descriptive  variables)  in  a sense  comparable  to  "social 
indicators"  and  will  become  part  of  what  will  be  known  as 
"basic  data"  in  future  phases.  To  what  extent  can  this  be 
anticipated,  and  what  difference  will  this  make  during  this 
phase? 

VI. 1.5  Discuss  your  understanding  of  the  special  requirements, 
if  any,  which  a government  agency  (or  its  contractor)  must 
meet  before  use  of  instruments  for  data  collection. 

VI.  1.6  What  advantages  or  disadvantages  are  introduced  by 
relevant  or  related  existing  data  banks  in  terms  of  the 
instruments  they  use  (e.g.,  various  definitions  of  industry 
classifications,  kinds  of  technology,  etc.)? 


VI . 


VI . 1 
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VI. 2 Information  Sources 

This  sub-factor  is  a measure  of  the  bidder's 
ability  to  identify  the  various  information 
sources,  and  to  gain  access. 

VI. 2.1  Discuss  the  important  or  critical  problems  you  anticipate 
in  this  area  and  your  capability  to  deal  with  them. 

VI. 2. 2 Discuss  the  various  strategies  for  identifying  sources 
of  information.  Distinguish,  if  appropriate,  between  sources 
of  information  needed  for  evaluating  a specific  experiment 
with  these  to  be  used  for  a broad  descriptive  (case)  study. 

VI. 2. 3 Discuss  the  issues  of  confidentiality  and  privacy  with 
respect  to  information  obtained  from  individuals.  Discuss 
the  issues  related  to  information,  in  general,  with  restrictions 
on  dissemination,  e.g.,  proprietary  information,  certain 
types  of  personnel  and  financial  information. 

VI. 2. 4 Discuss  the  issues  involved  in  access  to  individuals 
and  to  records.  A special  issue  of  concern  is  the  demands 
on  the  time  of  individuals  in  the  program  administrative 
agency,  and  others  who  may  have  a considerably  less  direct 
interest  in  the  evaluation. 

VI. 2. 5 Much  background  and  parametric  information  may  be 
available  in  some  form,  in  existing  records  which  have  been 
collected  for  some  other  purpose;  how  will  you  identify  these 
and  what  advantages  or  disadvantages  do  they  present? 
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VI . 3 Data  Collection 

/ 

This  sub-factor  is  measure  of  the  bidder's  capability 
to  organize,  staff,  train,  direct  and  control  the 
personnel  who  obtain  the  data  by  survey,  interview, 
observation,  or  use  of  records. 

VI. 3.1  It  is  anticipated  that  the  several  requirements  for 
data  collection  may  require  different  skills,  different 
organization,  and  different  methods  of  supervision  and 
control.  Discuss  by  example  or  reference  to  prior  experience 
your  capability  in  this  regard. 

VI.  3. 2 With  reference  to  the  question  asked  in  VI. 1.4, 
what  difference  in  approach  would  be  warranted  with  respect 
to  "basic  data"? 

VI. 3. 3 Discuss  or  describe  the  process  for  handling  data 
collection  (e.g.,  flowcharting).  Discuss  problems  of 
monitoring  and  evaluating  data  quality,  of  privacy  and 
confidentiality,  of  machine  acceptable  versus  non-machine 
acceptable  data  (e.g.,  unstructured  interviews). 
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VI . 4 Phase  Two  Data  Collection 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  anticipate  the  requirements  of  the  Phase  Two 
collection  of  both  basic  (base  line)  and  special 
data . 

VI. 4.1  It  is  proposed  that  the  bidder  will  be  able  to 
institutionalize  the  collection  of  basic  data  (i.e.,  key 
variables  and  parameters  common  to  a number  of  present 
or  expected  experiments,  or  key  exploratory  or  descriptive 
variables)  during  this  phase;  from  this,  base  line  data 
can  be  obtained  and  a "test"  of  the  process  can  be 
accomplished.  Comment  on  this. 

VI. 4. 2 It  is  proposed  that  there  will  remain  other  require- 
ments not  included  within  the  above  (e.g.,  specific  one-time 
requirements,  unusual  or  specialized  requirements;  added  or 
revised  requirements)  which  cannot  or  should  not  be  com- 
mingled with  the  above.  Discuss  this. 
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VI . 5 Phase  Three  Data  Collection 

i 

This  sub-factor  is  a measure  of  the  bidder's  ability 
to  plan  against  the  requirement  to  design  an 
ongoing  collection  process. 

VI . 5 . 1 At  least  the  basic  data  collection  process  should 
be  sufficiently  stabilized  and  described  to  allow  an 
ongoing,  turnkey  implementation.  Discuss  the  problems 
and  prospects  of  this. 

VI . 5 . 2 To  what  extent  can  new  or  other  specialized  require- 
ments be  anticipated? 

VI. 5. 3 Discuss  any  special  problems  with  respect  to  access, 
privacy,  confidentiality,  etc.,  which  may  be  introduced  when 
an  agency  takes  over  management  and  operation  of  the  evalua- 
tion system. 
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VII.  Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capability 
to  carry  out  all  of  the  functions  necessary  to  the 
analysis  of  the  data  and  to  present  the  results  and 
recommendations . 


VII. 1 Analysis 

This  sub-factor  is  the  measure  of  the  bidder's  ability 
to  identify  and  apply  the  appropriate  analytical 
(including  statistical)  techniques  to  the  data 
collected . 


VII.  1.1  Discuss,  by  example  or  reference  to  prior  experience, 
your  ability  to  accomplish  the  various  analytical  tasks 
involved . 

VII.  1.2  For  at  least  some  experiments  (and/or  policy  changes), 
a significant  if  not  primary  objective  is  economic.  Discuss 
your  capability  in  analysing  "cost/benefits,"  including 
problems  introduced  by  estimating,  allocating,  and  fore- 
casting costs  and  benefits,  both  direct  and  indirect. 

VII. 1.3  For  at  least  some  experiments  (and/or  policy  changes), 
a significant  if  not  primary  objective  is  to  bring  about 
some  change  in  the  policies  and  procedures  of  either  the 
program  administrative  agency  or  of  segments  of  the  industry. 
Discuss  your  capability  in  analysing  administrative 
(behavioral)  changes. 

VII. 1.4  For  at  least  some  experiments  (and/or  policy  changes), 
a significant  if  not  primary  objective  is  to  bring  about  some 
change  in  the  use  of  technology  - change  in  the  rate,  appli- 
cation, etc.  Discuss  your  capability  an  analysing  technolo- 
gical change. 
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VII. 2 Exploratory  (and/or  Descriptive)  Findings 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  exploratory 
findings . 


VII. 2.1  It  is  anticipated  that  in  many  areas  of  the  evalu- 
ation, it  will  be  neither  required  nor  feasible  to  develop 
findings  which  meet  the  more  stringent  formal  requirements 
of  a priori  proposition  testing.  To  what  extent  do  you 
consider  it  your  responsibility  to  reexamine  cases  where 
there  is  no  requirement,  especially  where  it  appears  that 
a priori  proposition  testing  i_s  feasible? 

VII. 2. 2 You  may  wish  to  comment  on  some  of  the  requirements 
as  presently  described  in  the  several  reference  documents. 

VII. 2. 3 You  may  wish  to  discuss  the  various  forms  of 
presentation  (i.e.,  case  studies,  informal  reports, 
status  reports,  systems  studies,  etc.) 

VII.  2. 4 In  addition  to  exploratory  (and/or  descriptive)  findings 
with  respect  to  experiments  (and  their  related  parameters  and 
background) , it  is  proposed  that  the  evaluation  system  design 
is,  itself,  an  exploratory  or  descriptive  "finding."  What 
differences,  if  any,  will  be  required  in  the  presentation  and 
related  supporting  documentation  or  justification  of  the 
evaluation  system  design? 
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VII. 3 A Priori  Proposition  Testing  Findings. 

This  sub-factor  is  a measure  of  the  bidder's 
capability  to  identify  and  present  a priori 
findings . 

VII. 3.1  It  is  anticipated  that  not  only  specific  experiments 
(at  least  in  terms  of  their  proximal  effects  or  impacts)  but 
also  other  specific  questions  which  may  be  subject 
to  study  or  experiment  will  require  (and  it  will  be  feasible 
to  accomplish)  a priori  proposition  testing  evaluation.  To 
what  extent  do  you  consider  it  your  responsibility  to  re- 
examine cases  where  there  is  no  requirement?  Cases  where 
there  is  a requirement,  but  it  is  not  feasible? 

VII. 3. 2 You  may  wish  to  comment  on  some  of  the  requirements 
as  presently  described  in  the  several  reference  documents. 

VII. 3. 3 You  may  wish  to  discuss  the  various  forms  of 
presentation. 

VII. 3. 4 Do  you  think  it  is  desirable  and/or  feasible  to  develop 
a procedure  or  process  which  can  be  used  to  make  preliminary 
assessments  of  the  likelihood  that  a particular  proposed 
experiment  will  be  capable  of  a priori  proposition  testing? 
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VII. 4 Recommendations 

This  sub-factor  is  a measure  of  the  bidder's 
understanding  of  the  relationship  between  the 
results  of  Phase  One  and  the  work  in  Phase  Two, 
in  terms  of  additional,  revised,  or  deleted 
hypotheses,  changes  in  objectives  or  background 
information,  and  the  relative  feasibility  and/or 
usefulness  of  alternative  designs  for  evaluation. 


VII. 4.1  In  addition  to  the  specific  findings  discussed  in 
VII. 2 and  VII. 3,  it  is  anticipated  that  the  work  in  each 
phase  will  provide  a basis  for  the  work  in  subsequent 
phases.  It  is  anticipated  that  this  input  to  the  next 
phase  may  become  available  before  the  completion  of  all 
of  the  work  in  the  current  phase.  Comment  on  the  feasibility 
of  an  overlap,  the  advantages  and  disadvantages. 

VII. 4. 2 It  is  anticipated  that  a "final  report"  will  be 
prepared  at,  or  after,  the  completion  of  each  phase  which 
will  describe  what  the  bidder  sets  out  to  do,  what  he  did, 
including  relevant  activities  of  others,  and  the  results 
(VII. 2,  VII. 3,  VII. 4).  Comment  on  the  relation  of  the 
final  report  to  the  several  other  forms  (and  timings)  of 
outputs . 
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I. 


II. 


Ill 


IV. 


V. 


VI. 


VII 


NUMERICAL  WEIGHTS  FOR  PROPOSAL  EVALUATION  FACTORS 


Factors  Weights 

Management 

This  factor  is  a measure  of  the  background 

and  overall  management  capability  of  the 

bidder.  15 

Reports  and  Reviews 

This  factor  is  the  measure  of  the  bidder's 
ability  to  assure  that  the  Government  is 
able  to  effectively  and  progressively  review 
a very  large  scale  effort  with  a minimum  of 
people,  and  of  the  bidder's  ability  to 
coordinate  his  activities  with  those  of 
other  related  individuals  and  organizations.  15 

Obj  ectives 

This  factor  is  a measure  of  the  bidder's  over- 
all understanding  of  the  objectives  of  this 
procurement  (and  related  programs) . 20 

Background  Information 

This  factor  is  a measure  of  the  bidder's  over- 
all understanding  of  the  background  and 
characteristics  of  this  procurement.  15 

Evaluation  Design 

This  factor  is  a measure  of  the  bidder's 
capabilities  to  carry  out  the  evaluation 
design  requirements.  15 

Data  Collection  Process 

This  factor  is  a measure  of  the  bidder's  capa- 
bility to  carry  out  all  of  the  functions 
necessary  to  the  acquisition  of  data  on  the 
variables  and  parameters  of  interest.  10 

Data  Analysis  Process 

This  factor  is  a measure  of  the  bidder's  capa- 
bility to  carry  out  all  of  the  functions 
necessary  to  the  analysis  of  the  data  and  to 
present  the  results  and  recommendations.  10 


Seven  Evaluation  Factors  Total  100 
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SCOPE  OF  EFFORT 
Introduction 


A.  Purpose 

The  purpose  of  this  document  is  to  provide  a guide  to  the 
bidder  in  the  preparation  of  supporting  data  on  the  scope 
of  effort  he  proposes. 

B.  Basis  for  Preparation  of  This  Document 

This  document  is  intended  to  be  used  in  conjunction  with  the 
associated  Proposal  Preparation  Procedure,  Proposal  Evaluation 
Factors,  Schedule,  and  Statement  of  Work.  The  objective  is  to 
assist  the  bidder  in  presenting  his  supporting  data  on  scope 
of  effort  in  an  orderly  and  usable  form  so  that  it  may  be 
evaluated  concurrently  with  his  written  (and  oral)  proposal. 

B.l  In  order  to  achieve  an  orderly  and  usable  form,  the  sup- 
porting data  on  scope  of  effort  must  be,  at  least  substantially, 
related  to  the  factor  (and  sub-factor)  categories  which  are  the 
basis  for  the  evaluation.  The  detailed  outline  furnished  in 
the  next  section  is  for  this  purpose. 

B.2  In  addition  to  the  above,  it  is  necessary  to  assure  that 
the  data  furnished  by  the  several  bidders  be  on  a reasonably 
comparable  basis.  To  accomplish  this  the  detailed  guidance 
included  in  this  Introductory  Section  is  furnished.  It  should 
be  understood  that  the  guidance  in  this  respect  is  necessarily 
arbitrary  in  the  interests  of  meeting  the  specific  objective. 

B.3  The  format  requirements  reflect  the  fact  that  evaluation 
team  members  need,  and  can  use,  only  selected  data,  primarily 
in  terms  of  man/months  of  effort  in  certain  areas  and  certain 
materials  costs.  A summary  analysis  of  the  scope  of  effort 
against  cost  data  will  be  accomplished. 
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C . General  Comments 

To  assist  the  bidder  in  understanding  more  fully  the  part 
which  these  data  will  have  in  the  evaluation,  these  general 
comments  are  offered. 

C.l  This  type  of  program  which  establishes  a broad  objective 
but  provides  for  planning  and  design  to  work  out  with  more 
detail  and  precision  the  means  for  accomplishment  cannot  be 
costed  with  the  closeness  normally  experienced  on  procurements 
of  items  with  a stabilized  design.  For  this  reason  the 
bidders'  respective  total  cost  estimates  will  not  necessarily 
reflect  the  actual  realized  costs  to  the  government,  nor  do 
they  necessarily  reflect  the  relative  costs  of  the  bidders 
in  providing  a comparable  capability.  Therefore,  relative 
total  cost,  as  such,  cannot  and  will  not  be  given  any  direct 
weight  in  the  evaluation.  Cost,  in  terms  of  scope  of  effort, 
however,  will  aid  in  establishing  the  kind  and  extent  of  effort, 
and  this  information  will  serve  to  further  define  the  "word- 
pictures"  included  in  the  written  (and  oral)  proposals.  The 
formal  evaluation  of  relative  cost  and  associated  areas  is 
included  as  part  of  one  of  the  major  factors  in  the  evaluation 
(1.4),  and  further  reference  should  be  directed  there. 

C.2  While  the  major  impact  of  these  data  in  the  evaluation  will 
be  in  providing  additional  perspective  to  the  written  (and 
oral)  proposals,  certain  corollary  information  will  be  used. 

C.2.1  A measure  of  how  realistically  and  thoroughly  the  costing 
has  been  done  will  be  established.  A too  high  estimate  indi- 
cates the  obvious  disadvantages  of  "gold  plating"  or  a 
potentially  loose  operation.  In  contrast,  however,  a too  low 
estimate  presents,  in  the  long  run,  even  more  serious  dis- 
advantages. First,  it  may  indicate  a serious  misunderstanding 
of  the  scope  and  objective  of  the  program.  Second,  it  may 
raise  a serious  question  as  to  the  bidder's  ability  to  plan 
and  estimate  costs,  thus  impairing  his  ability,  and  that  of 
the  government,  to  review,  predict  and  control  costs.  Finally, 
it  may  indicate  something  less  than  complete  candor  in  presenting 
the  cost  part  of  the  proposal. 
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C.2.2  The  data  presented  here  will  be  used  to  supplement  the 
data  furnished  with  respect  to  cost  as  a factor  in  the  over- 
all evaluation. 

C.2.3  The  data  presented  here  will  be  used  in  negotiation  with 
the  successful  bidder  both  as  a basis  for  further  refining  the 
contractual  statement  of  work  and  in  establishing  the  contract 
price . 
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D.  Use  of  Format 


D.l  It  is  of  critical  importance  that  all  persons  directly  or 
indirectly  preparing  data  on  scope  of  effort  be  thoroughly 
familiar  with  the  rules  on  use  of  the  format.  This  is 
especially  true  because  the  figures  to  be  included  do  not, 
in  all  cases,  follow  the  accounting  distinction  between  direct 
and  indirect  costs.  Further,  certain  types  of  effort  are  to 
be  tabulated  as  separate  items.  An  effort  has  been  made  to 
use  format  and  terminology  as  close  to  the  general  practice 
as  possible.  Where  this  has  not  been  done,  the  basis  has 
been  to  assume  that  useful  data  for  the  evaluation  can  be 
obtained.  Certain  distinctions,  such  as  direct  and  indirect 
costs,  have  little  significance  for  this  purpose.  Similarly, 
certain  areas  of  effort  can  only  be  evaluated  if  separated 
from  proportionately  large  associated  areas  of  effort. 

D.2  The  next  section  specifies  the  scope  of  effort  data  on 
a factor  by  factor  basis.  To  facilitate  the  evaluation,  these 
data  should  be  collated  with  the  respective  factors.  Because 
the  procedure  for  evaluation  provides  for  each  factor  to  be 
evaluated,  substantially,  without  mandatory  recourse  to  other 
parts  of  the  proposal,  it  is  critically  important  that  the 
data  on  scope  of  effort  and  the  written  proposal  on  a specific 
factor,  taken  together,  is  complete  and  self-explanatory.  Care 
must  be  exercised  to  assure  that  the  evaluation  team  does  not 
inadvertently  penalize  the  written  proposal  because  of  an 
inadequate  scope  of  effort. 

D.2.1  Under  each  factor,  the  scope  of  effort  should  be  set 
out  separately  for  the  first  phase  of  the  program.  Where 
the  bidder  proposes  to  begin  or  complete  items  in  another 
phase,  note  should  be  made  to  avoid  misunderstanding. 

D.3  It  is  expected  that  in  many  instances  the  scope  of  effort 
proposed  will  be,  literally,  the  bidder's  present  estimate 
of  the  approximate  level  (or  scope)  of  effort  he  expects 
will  be  required  to  accomplish  what  he  proposes  to  do  in 
his  written  (and  oral)  proposal.  A certain  latitude  is 
expected,  and  these  data  on  scope  of  effort  will  be  used, 
as  stated  before,  primarily  to  assist  the  evaluation  team 
in  understanding  what  the  bidder  is  actually  proposing. 

There  are,  however,  certain  areas  which  may  prevent  a clear 
understanding,  and  some  of  these  are  tabulated  below. 
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D.3.1  The  breakdown  in  the  next  section  is  a minimum  guide 
in  tabulating  the  scope  of  effort.  Where  a particular  figure 
is  considered  unusually  small  or  large,  or  for  some  other 
reason  may  be  misleading,  it  should  be  footnoted  with  an 
explanation  or  further  breakdown. 

D.3.2  In  some  cases  identification  of  particular  types  of 
effort  with  a proposed  "associate"  or  subcontractor  will  be 
of  assistance.  This  will  be  particularly  true  where  part  or 
all  of  an  effort  is  proposed  to  be  accomplished  by  some 
specialist  group. 

D.3.3  The  analysis  of  comparative  cost  advantage  will  not  be 
accomplished,  as  a separate  factor,  by  the  overall  evaluation 
team.  It  is  necessary,  however,  where  a bidder  is  estimating 
costs  which  are,  on  a relative  basis,  low  that  the  basis  for 
this  be  included  in  the  scope  of  effort  data  to  assure  that 
the  evaluation  team  understands  that  the  bidder  does  intend 
to  accomplish  what  he  has  said  in  his  written  proposal. 

D.3.4  Conversely,  where  a possible  misunderstanding  may  occur 
due  to  a lack  of  appreciation  by  the  evaluating  team  of  the 
cost  associated  with  accomplishing  a particular  part  of  the 
proposal,  the  bidder  should  make  this  clear. 

D.3.5  Man/month  estimates  should  be  divided  according  to 
distinct  classes  to  avoid  confusion  or  ambiguity  in  two 
respects.  First,  the  bidder's  data  should  distinguish  between 
direct  and  indirect.  Second,  the  data  should  distinguish 
between  senior  professional  personnel  and  supporting  technicians, 
draftsmen,  etc.,  and  between  various  management  personnel  and 
clerical  assistance,  etc. 

D.3.6  Any  further  analysis  considered  necessary  to  assure  that 
the  evaluation  team  properly  assesses  the  relative  scope  of 
effort  as  related  to  the  written  proposal  is  encouraged.  This 
may  be  particularly  true  in  terms  of  level  of  training  of 
personnel,  availability  of  facilities,  ability  to  control  design, 
and  the  like. 
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E . Summary  Data 

E.l  The  relationship  of  the  individual  scope  of  effort  on 
the  several  factors  to  the  bidder's  overall  cost  estimate  will 
be  examined.  For  this  purpose,  certain  summary  data  will 
be  required.  These  summary  data  should  be  furnished  as  a 
separate  document  which  should  also  include  a complete  reference 
set  of  the  scope  of  effort  data  sheets. 

E.2  Basic  summary  data  will  include  presentation  of  sub-total 
and  total  cost  estimates  in  standard  format  - direct  costs, 
burden,  materials,  G&A,  fee,  etc. 

E.2.1  These  summary  sheets  should  be  accompanied  with  notes 
reconciling  the  scope  of  effort  data  sheets.  Any  unusual 
data  should  be  explained. 

E.3  If  the  proposal  includes  a substantial  proposed  sub- 
contracting (or  "associate")  cost  figures,  a summary  analysis 
should  be  prepared  and  related  to  the  basic  summary  data. 

E . 4 Supporting  schedules  should  be  prepared  detailing  the 
hourly  wage  rate  of  personnel  and  this  related  to  the  basic 
summary  data. 

E.5  Supporting  schedules  should  be  prepared  for  travel  costs 
and  related  to  the  basic  summary  data.  Schedules  should 
include  breakdown  based  on  distance  (or  destination) , duration, 
number  of  travelers,  and  number  of  trips.  Similarly,  supporting 
schedules  on  telephone  (and  teletype)  by  number  of  calls,  and 
average  cost  per  call. 


82 


RFP  6-35756 

Phase  One  Scope  of  Effort 

Note:  The  detailed  organization  parallels  the  organization 

of  the  Statement  of  Work,  Schedule (s),  and  Proposal 
Evaluation  Factors,  reference  to  which  should  be  made 
for  clarif ication . ' 

I.  Management 

(Note  that  the  scope  of  work  here  is  limited  to  admini- 
strative or  program  management,  and  should  be  distinguished 
from  effort  properly  reported  under  other  factors.) 

Program  Director 

Senior  Management  Advisors 

(Include  general  officers,  or  other  senior 
corporate  personnel,  concerned  with  admini- 
strative management  policy,  if  appropriate) 

Other  (specify) 

(Include  comparable  personnel  of  associates, 
subcontractors,  consultants,  if  appropriate; 
include  program  director's  personal  staff, 
if  any) 

Scheduling 

(Include  costs  of  analysis,  preparation, 
monitoring,  etc.) 

Costing 

(Include  costs  of  analysis,  preparation, 
monitoring,  etc.) 

Telephone 

(Include  telephone,  teletype  costs,  etc.,  and 
breakdown  by  number  of  calls,  etc.,  if  appropriate) 

Travel  $ 

(Include  breakdown  by  destination,  duration, 
breakdown  by  number  of  trips,  if  appropriate) 

II.  Reports  and  Reviews 

(Note  that  you  should  distinguish  efforts  properly 
attributable  to  other  factors.) 

Liaison  with  ETIP  M/M 

(Include  time  spent  in  conferences  or  visits  both 
at  ETIP  and  contractor's  plant,  or  elsewhere  break- 
down by  number  of  trips,  duration,  number  of 
travelers . ) 

Liaison  with  PAA  M/M 

(Include  on  same  basis.) 

Liaison  with  Others  M/M 

(Include  on  same  basis.) 


M/M 

M/M 

M/M 

M/M 

M/M 

$ 
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Reports  and  Services  M/M  & $ 

(Include  time  spent  in  direct  preparation, 
technical  writing,  drafting,  etc.,  and 
costs  of  reproduction.  Breakdown  by  type 
of  report,  etc.,  if  appropriate.) 


III.  Objectives 

Review  of  objectives  M/M 

(Breakdown  between  documents  and  interviews, 
and  among  ETIP,  PAA,  industry,  and  others.) 

Preparation  of  description  and  analysis  M/M 

(Breakdown  among  ETIP,  PAA,  industry,  and 
others . ) 

Preparation  of  preliminary  recommendations  M/M 

Preliminary  requirements  for  pilot  evaluation  M/M 

system 

Preliminary  requirements  for  prototype  M/M 

evaluation  system 

Travel  $ 

IV.  Background 

Review  of  background  M/M 


(Breakdown  between  documents 

and  interviews,  and  among  experiments, 

programs  and  organizations.) 


Preparation  of  framework  model  M/M 

Travel  $ 

V.  Evaluation  Design 

Preliminary  Design  M/M 

Detailed  Design  M/M 

Preliminary  design  of  pilot  evaluation  system  M/M 


Preliminary  design  of  prototype  evaluation  system  M/M 

VI.  Data  Collection  Process 

(Note  that  you  should  distinguish  data  collections 
which  may  have  been  included  under  prior  factors) 

Design  and  development  of  data  collection  process  M/M 

Data  collection  for  exploratory  evaluations  M/M 
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Design  and  pilot  test  of  instruments  M/M 

Data  collection  for  a priori  proposition  M/M 

testing  evaluations 

Travel  $ 

(Breakdown,  as  appropriate,  between  exploratory 
and  a priori  proposition  testing  evaluations; 
provide,  as  appropriate,  basis  for  estimating 
in  terms  of  trips,  etc.) 

Telephone,  postage,  etc.  $ 

(Breakdown,  as  above.) 

VII.  Data  Analysis  Process 

(Note  that  you  should  distinguish  from  effort 
included  with  Factor  II.) 

Design  and  development  of  data  analysis  process  M/M 

Exploratory  analysis  and  findings  M/M 

A priori  proposition  testing  analysis  and  findings  M/M 
Analysis  of  evaluation  system,  and  recommendations  M/M 
Analysis  of  Phase  One,  and  recommendations  M/M 
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Phase  Two  and  Three  Scope  of  Effort 

Phase  Two 

It  is  not  expected  that  the  bidder  will  be  able  to  provide  a 
comparable  level  of  detail  for  Phase  Two.  However,  a rough 
estimate  will  be  useful  in  assessing  those  references  to 
Phase  Two  in  the  proposal,  and  in  planning. 

The  minimum  required  breakdown  is  to  distinguish  between  the 
following: 

1.  Those  costs  (labor,  travel,  etc.)  directly  a function  of 
data  collection  and  analysis  for  both  basic  and  special 
data  processes. 

2.  All  of  the  remainder  (management,  design,  etc.) 

Additional  information,  consistent  with  these  purposes, 
would  be  appreciated,  but  it  is  not  required. 


Phase  Three 

A similar  distinction  should  be  observed  in  providing  estimates  for 
Phase  Three.  It  is  expected  that  these  may  be  more  speculative, 
but  where  there  are  identifiable  assumptions  which  will  largely 
determine  the  scope  of  effort,  discussion  would  be  helpful. 
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